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Abstract. The goal of harmonic analysis on a (noncommutative) group is to de- 
compose the most "natural" unitary representations of this group (like the regular 
representation) on irreducible ones. The infinite-dimensional unitary group U{oo) is 
one of the basic examples of "big" groups whose irreducible representations depend 
on infinitely many parameters. Our aim is to explain what the harmonic analysis on 
U{oo) consists of. 

We deal with unitary representations of a reasonable class, which are in 1-1 corre- 
spondence with characters (central, positive definite, normalized functions on U{oo)). 
The decomposition of any representation of this class is described by a probability 
measure (called spectral measure) on the space of indecomposable characters. The 
indecomposable characters were found by Dan Voiculescu in 1976. 

The main result of the present paper consists in explicitly constructing a 4— 
parameter family of "natural" representations and computing their characters. We 
view these representations as a substitute of the nonexisting regular representation 
of ?7(oo). We state the problem of harmonic analysis on C7(cxd) as the problem of 
computing the spectral measures for these "natural" representations. A solution to 
this problem is given in the next paper [B04], joint with Alexei Borodin. 

We also prove a few auxiliary general results. In particular, it is proved that 
the spectral measure of any character of U{oo) can be approximated by a sequence 
of (discrete) spectral measures for the restrictions of the character to the compact 
unitary groups U{N). This fact is a starting point for computing spectral measures. 
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Introduction 



(a) Preface. The problem of noncommutative harmonic analysis consists in de- 
composing "natural" unitary representations of a given group into irreducible ones. 

For instance, if is a compact group, then the decomposition of the (bi)regular 
representation in the space L'^{K) is described by the classical Peter- Weyl theorem. 

Another well-known example is the decomposition of the quasiregular represen- 
tation of a noncompact simple Lie group G acting in the space on the Riemannian 
symmetric space G/K. 

Note that in the Pctcr-Weyl theorem, the spectrum of the decomposition is 
discrete, while for L'^{G/K) the spectrum is continuous. The decomposition of 
L'^(G/K) is described by a measure living on a region of dimension equal to the 
rank of the space G/K. 

The problems of decomposing the space on G and on a pseudo-Riemannian 
symmetric space G/H belong to the next levels of difficulty. 

In the present paper and the next one (joint with Alexei Borodin, [B04]), we 
deal with the problem of harmonic analysis in a totally different situation. The 
novelty is that the group is no longer compact or locally compact, its dual space 
has infinite dimension, and the decomposition into irreducibles is governed by a 
measure with infinite-dimensional support. 

Our main result (established in [B04]) is an explicit description of measures 
which arise in this way (we call them spectral measures). The description is given 
in the language of stochastic point processes. These probabilistic objects have never 
emerged in classical representation theory. However, they happen to be an adequate 
tool for groups with infinite-dimensional dual. 

The present paper contains results of two kinds: 

First, we construct a family of representations which we consider as "natural" 
ones. In our situation, when the group is not locally compact, the conventional 
definition of a regular representation (or a quasi-regular representation associated 
with a homogeneous space) is not applicable directly. This forces us to choose 
another, more sophisticated, way to produce representations. 

Second, we prove necessary general theorems concerning the spectral measures 
with infinite-dimensional support. 

This finally allows us to convert the problem of harmonic analysis to an asymp- 
totic problem of the form which is typical for random matrix theory or asymptotic 
combinatorics. We are lead, however, to a new model, which was not previously 
examined. 

We proceed now to a more detailed description of the contents of the present 
paper. 

(b) The group. Consider the chain of the compact classical groups U{N), N = 
1,2,..., which are embedded one into another in a natural way, and let U (oo) be 
their union. Equivalently, elements of U{oo) are infinite unitary matrices U = \Uij\i 
where the indices i,j take values 1,2,..., and we assume that Uij = 5ij for i + j 
large enough. The group U (oo) is one of the fundamental examples of inductive 
limit groups (another such example is <S'(oo), the union of the finite symmetric 
groups). 

Following the philosophy of [Oil], [013], we form a (G, i^)-pair, where G is the 
group C/(oo) X U{oo) and K is the diagonal subgroup in G, isomorphic to U{oo). 
This is a Gelfand pair in the sense of [013]. 
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(c) Representations and characters. We are dealing with unitary representa- 
tions T of the group G possessing a distinguished cychc K-invariant vector ^. Such 
representations are called spherical representations of the pair {G,K). They are 
completely determined by the corresponding matrix coefficients V'l") — 

The V^'s are called the spherical functions. These are certain JC-biinvariant func- 
tions on G, which can be converted (via restriction to the subgroup U (oo) x {e} C G) 
to certain central functions x on U{oo). The functions x thus obtained are called 
the characters of the group [/(oo). The correspondence T ^ x makes it possible 
to employ both languages, that of spherical representations and that of charac- 
ters, each of which has its own merits. To irreducible representations T correspond 
extreme characters x (i-e-, extreme points in the convex set of all characters). Irre- 
ducible spherical representations of {G, K) and extreme characters of U (oo) admit 
a complete description. They depend on countably many continuous parameters. 

(d) How to get "natural" representations. Set G{N) = U{N) x U{N) and let 
K{N) be the diagonal in G{N), N = 1,2,... . The homogeneous space G{N) /K{N) 
can be identified with the group space U{N), and then the action of G{N) on 
G{N)/K{N) turns into the two-sided action of U{N) on itself. Let Regjy denote 
the quasi-regular representation of G{N) in L'^{G{N) / K{N)). This is nothing else 
than the biregular representation of the compact group U (N) whose decomposition 
is determined by the Peter- Weyl theorem. 

We seek for a counterpart Reg^^ of the representations Reg^ for the pair (G, K). 
As was mentioned above, there is no good measure on G/K, hence no space 
1?{GIK). To overcome this difficulty, two general recipes are known. First, to em- 
bed representations Reg^v in each other and then to define Reg^^ as the inductive 
limit representation linj^^^Reg^. Second, to embed GjK into an appropriate 
G-spacc GjK possessing an invariant (or quasiinvariant) measure m and then to 
realize Reg^^ in iP'iG j K ,r(i). 

A realization of any of these recipes is by no means an automatic exercise: 

A subtle point of the inductive limit construction is that there are many different 
embeddings tAr : Regjv ~^ R^Sat+i for each N , and the limit representation Regg^ 
highly depends of the chain {ijv} chosen. Trying all the possible chains {iat} one 
gets too many limit representations, so that a fine selection rule must be imposed. 

As for the second way, we have to guess what the ambient space GjK should 
be. With GjK specified, we next have to find good measures m (and moreover, to 
select some 1-cocycles that are also involved in the construction). 

We employ both methods and finally get a family {Tz„} of representations de- 
pending on two complex parameters z,w such that 3?(2 + «;) > —\. We believe 
that the representations T^w are "natural" objects of harmonic analysis. 

The space GjK is constructed as follows. We define certain projections of the 
group spaces U (N) U{N — 1), where A?" = 2, 3, ... , and then take the projective 
limit space U = ^mC/(A"). This is our G/K. 

(e) Gelfand— Tsetlin graph and coherent systems. The Gelfand-Tsetlin graph 
is a convenient tool for writing characters of C/(oo). The vertices of the graph sym- 
bolize the irreducible representations of various groups U (N) while the edges code 
the inclusion relations between irreducible representations of U{N) and U{N + 1). 
By GTtv we denote the subset of vertices corresponding to irreducibles of U{N); 
elements of GTn are identified with dominant weights for U{N), i.e., these are 
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A'"-tuples A = (Ai > • • • > Ajv) of integers. 

Given a character x of U{oo), we can expand its restriction to the subgroup 
U{N) into a convex combination of the functions X^{')/x^{^)i where stands for 
the irreducible character (in the conventional sense) of U{N), indexed by A e GT^v- 
The coefficients Pat (A) of this expansion determine a probability distribution Pn 
on the discrete set GTn- 

In this way, we get a bijection x ^ {Pn}n=i,2,... between characters x ^iid 
certain sequences {Pn} of probability distributions. These sequences are called 
coherent systems, because, for any N, the distributions Pn and Pn+i are connected 
by a certain "coherency relation" . 

(f) The characters Xzw and their analytic continuation. Having constructed 
the representations Tzw, we proceed to the corresponding characters Xzw- We eval- 
uate them in terms of the associated coherent systems {Pn}- The expressions that 
we get for Pn make sense, via analytic continuation, for a larger set of parameters. 

Specifically, let z, z, w, w' be the coordinates in C^, and let V be the open halfs- 
pace in C"^ determined by the inequality ^{z + w + z' + w') > — 1. We exhibit an 
"admissible subset" X>adm C I> of real dimension 4, containing as a proper subset 
all the quadruples {z, z,w,w') G V with z' = z, w' = w, and such that for any 
quadruple {z, z,w,w') e "Dadm) the following formulas provide a coherent system: 

Piv(A I z, z, w, w') = {Sn{z, z, w, w'))"^ X r^--)2 — ~ 



X 



^ 1 

n 



Li + -K + + + 1 + Ai - i)T{w' + N + l + Xi-i)' 

where N = 1,2,..., A ranges over GTjv, and Sn{z, z,w,w') is a normalization 
constant: 

^ . /\_Y\ T{z + z' + w + w' + i) 

n[z, z,w,w)- II + ^ + i^Y{z + w' + i)r{z' + W + i)r{z' + w' + i)rii) ' 

Thus, to any (z, z, w, w') e I^adm, we may assign a character Xz,z,w,w'- The initial 
characters Xzw = Xz,w,z,w form the "principal series", and the remaining characters 
belong to its analytic continuation. The whole picture resembles the conventional 
principal, complementary and degenerate series for semi-simple Lie groups, so that 
we even employ this terminology. However, in our context, these concepts refer not 
to (generically) irreducible representations, as in the conventional context, but to 
highly reducible ones. 

Note that the construction of the coherent systems implies a curious summation 
formula, 

Pn{^ I z,z,w,w') = 1. 

Ai>-->A„ 
Ai , . . . , A jv 

In the simplest case = 1 it looks as 

E 



T{z-k + l)T{z' -k + l)T{w + k + l)T{w' + k + 1) 
_ T{z + z' + w + w' + 1) 

" r{z + w + i)r{z + w' + i)r{z' + w + i)r{z' + w' + i) 
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which is equivalent to a classical identity due to Dougall, see [AAR, Chapter 2, 
Theorem 2.8.2 and Exercise 42(b)], [Er, §1.4]. 

(g) Abstract theorems on spectral measures. Let us return now to extreme 
characters. There is a bijective correspondence x^'^'' ^ ^ between extreme charac- 
ters and points uj of an infinite-dimensional "region" 

O C m4~+2 = M~ X M~ X M~ X X R X M. 

For the functions X^^HU), where U G U{oo), there is a remarkable explicit formula 
due to Voiculescu (see (1.2) below). 

Our first "abstract" theorem says that for any character here exists a unique 
probability measure P on Q such that 

X{U)^ [ x^^\U)P{duj), UeU{oo). 
Jo. 

We call P the spectral measure of x- Conversely, any probability measure P on 
is a spectral measure for a certain x- This result is a refinement of a theorem 
due to Voiculescu [Vo]. It provides a nice general description of the whole set of 

characters. 

The next result shows that spectral measures can be, in principle, computed. 
Specifically, we define, for any N = 1,2,..., an embedding GTjv ^ ^ such that 
the image of GTjv looks as a discrete approximation of 17, which becomes more and 
more exact as ^ oo. 

Now let X be an arbitrary character, {-Pat} be the corresponding coherent system, 
and P be the spectral measure of x- Then, according to the second "abstract" 
theorem, we have P = \imN-,oo Pn, where we identify Pn with its pushforward 
under the embedding GTjv ^ 0,. 

(h) The problem of harmonic analysis on U{oo). Now we are in a position 
to state this problem explicitly: 

Let X = Xz,z,w,w'i where {z, z, w, w') G I^adm, and let {-Pat} be the corresponding 
coherent system. Recall that P^ = Pn{ ' \ z,z,w,w') is a probability measure 
on GTjv, given by the explicit formula above. Let us carry over Pn to the space 
fl. Then the problem consists in evaluating the limit of the measures Pn in the 
ambient space as AT — > oo. 

(By virtue of the "abstract" theorems above, the limit always exists and coincides 
with the spectral measure.) 

The solution to the problem is presented in the next paper, [B04]. 

(i) Connections with infinite random matrices [B03]. Let Sj be the space of 
all infinite Hermitian matrices and let U{oo) act on by conjugations. There is a 
parallelism: 



Characters x oi U{oo). 
Extreme characters x^^\ 
indexed by points uj 
of a region n C M^°°+2. 
Decomposition x = /o X^'^^ Pidi^)-) 
where P is a probability measure 



Invariant probability measures M on i^. 
Ergodic invariant measures M^^\ 
indexed by points cu 
of a region n C ^^00+2 
Decomposition M = j^M^"^^ P{(ko), 
where P is a probability measure 
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on J] C 

Distinguished characters x = Xz-. 
where z, w G C, ^{z + w) > — |. 
The problem of computing 
the spectral measures P 
for distinguished characters x- 



Distinguished invariant measures 

where s G C, > — |. 

The problem of computing 

the spectral measures P 

for distinguished measures M. 



And so on. 

In the paper [BOS], we constructed the distinguished measures m^^^ and com- 
puted their spectral measures. The whole theory of ?7(oo)-invariant measures can 
be viewed as a simplified version of the theory of characters. 

On the other hand, the results of [B03] are directly used in the present paper: 
the space it mentioned above in subsection (d) can be identified (within a negligible 
subset) with the space S), and the measures m involved in the construction of the 
representations T^^ are nothing else than the measures m^^^ . 

(j) Characters of S{oo). As was mentioned above, the group S{oo), the union of 
finite symmetric groups S{n), is another fundamental example of an inductive limit 
group. The representation theories of the both groups, U{oo) and S{oo), reveal 
deep analogies and links. Of course, in the classical representation theory, it is well 
known that representations of the groups S{n) and U{N) are connected by the 
Schur-Weyl duality. But in the case of infinite dimension the connections between 
the symmetric and unitary groups turn out to be much closer. 

The problem of harmonic analysis on S{oo) was stated in [KOV] and then further 
developed in a cycle of papers [P.I-V], [Borl-2], [BOl-2]. The construction [KOV] 
of the "generalized regular representations" served as a guiding example for 
the construction of the representations T^^ in the present paper. Further, the 
experience of our work on the spectral measures for S{oo) helped us very much in 
the work [B04]. 

(k) Generalization to other {G, JC)— pairs. Works of Pickrell [Pi] and 
Neretin [Ner3]. The (G, K)-pair (^7(00) x [/(oo), diagonal C/( 00)) is a representa- 
tive of the family of ten (G, iir)-pairs, which come from the ten classical series of 
compact Riemannian symmetric spaces. This family is a natural framework for de- 
veloping representation theory (see [Oil], [013], [Nerl]) and, in particular, harmonic 
analysis. In the pioneer work [Pi], Pickrell considered the pair G = lin^U{2N), 
K = U{N) X U{N), which corresponds to the series of complex Grassmanians 
U{2N) /U{N) X U{N). He constructed the ambient space G/K as a projective 
limit of Grassmanians and defined a family of measures m which give rise to "nat- 
ural" representations. Pickrell's construction was the starting point of [KOV] and 
of the present paper. In the recent paper by Neretin, [Ner3], Pickrell's construction 
is carried over to all ten pairs. 

Note that, as compared with Pickrell's results, our construction of the represen- 
tations Tziu incorporates a few new observations. 

First, following [KOV], we introduce a complex parameter instead of a real one 
(the idea is to employ a wider family of cocycles for an action of the group G) . It 
is worth noting that the same generalization makes sense for all ten pairs (G, K) 
mentioned above. 

Second, for our pair (G, K), it is actually possible to introduce two complex 
parameters z, w. This (nonevident) fact was prompted by Neretin's results on 



Hua-type integrals, see [Ner2], [Ner3]. 

Third, we observe that the construction of the representations Tzw makes sense 
for all 2:, ti; G C But for "Rz + "Rw < —1/2 wc get representations without a distin- 
guished iiT-invariant vector. It would be interesting to study these representations. 

(1) Acknowledgment. This paper is part of a joint project with Alexei Borodin. 
I am very grateful to him and also to Sergei Kerov, Yurii Neretin, and Anatoly 
Vershik for numerous discussions. 

1. Characters 

Definition 1.1. Let K be a topological group. By a character of K we mean any 
continuous complex-valued function x on satisfying the following three condi- 
tions: 

(i) X is central, i.e., constant on conjugacy classes; 

(ii) X is positive definite, i.e., for any finite collection gi, ■ ■ ■ , Qn of elements of 
K, the n X n matrix [xioj^ 9i)]i<i,j<n is Hermitian and nonnegative; 

(iii) X is normalized at the unit element, i.e., x(e) = 1. 

Let X{K) denote the set of the characters of K. Evidently, X{K) is a convex 
set. Its extreme points are called extreme (or indecomposable) characters. -"^ 

Example 1.2. Let be a finite group or, more generally, a compact separable 
group, and let K be its dual space, i.e., the set of equivalence classes of irreducible 
representations. Then ii' is a finite or a countably infinite set. Let A range over 
K and x^ denote the irreducible character corresponding to A (i.e., the trace of an 
irreducible representation in the class A). The extreme characters of K in the sense 
of Definition 1.1 are exactly the normalized irreducible characters 

X\9) = ^y AGK, (1.1) 
while general characters x ^ '^(-^) ^^'^ convex combinations of the extreme ones. 

Thus, the set X{K) is a simplex with vertices indexed by A e 

In the example above the set X{K) is large enough to separate conjugacy classes. 
If K is not compact it may well happen that X{K) is very small (e.g., is exhausted 
by the function x = 1). Actually, the class of groups K with X[K) large enough is 
rather restricted. However, it includes important examples leading to a rich theory. 

In the present paper we take as K the infinite-dimensional unitary group, which 
is defined as 

C/(oo) = U U{N), 

N>1 

where U{N) is the group of N x N unitary matrices. The embedding U{N) 
U{N + 1) is defined as follows: we identify U{N) with the subgroup in U{N + 1) 

^This terminology differs from that used in [Till], [Th2], [VKl], [VK2], [Vo]. In those papers, 
only extreme characters were considered, and they were called simply characters. 
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fixing the (A'" + l)st basis vector. Equivalently, U{oo) is the group of infinite unitary 
matrices U = [Uij], i,j = 1, 2, ... , with finitely many matrix entries Uij distinct 
from 5ij. 

We equip f/(oo) with the inductive Umit topology (i.e., a function on [/(oo) is 
continuous if its restriction to each subgroup U{N) is continuous). Note that U{oo) 
is not locally compact. 

First, let us describe the conjugacy classes in U{oo). Recall that the conjugacy 
classes in U{N) are parameterized by spectra of unitary matrices, that is, by un- 
ordered A^-tuplcs Ui, . . . , tijv of complex numbers with modulus 1. In this notation, 
the embedding U{N) ^ U{N + 1) is described as (wi, . . . , u„) i— > (wi, . . . , w^v, !)• 
Thus, the conjugacy classes in U{oo) can be parameterized by countable collec- 
tions of complex numbers {ui,U2, ■ ■ ■) such that \ui\ = 1 and only finitely many 
of Ui^s are different from 1; the ordering of w^'s is unessential. As a representa- 
tive of a conjugacy class indexed by («i, U2, ■ ■ ■) one can take the diagonal matrix 
diag(Mi, W2, ■ ■ ■)• 

To describe the extreme characters of U{oo) we need some notation. 
Let denote the product of countably many copies of M, and set 

]R4oo+2 ^ X X X M°° X M X R. 
Let n c be the subset of sextuples 

such that 

q;± = (af > > • • • > 0) e ]R°°, = (/3^ > > • • • > 0) e M°°, 

oo 

Set 

oo 
i=l 

and note that 7"'~,7~ are nonnegative. 

To any a; e we assign a function X^'^) on U{oo): 

n W''---'----''n i:gi::;ii:s::::;; l ■ 

ueSpectrum(U) I i=l ^ ^> ^ ^> ) 

(1-2) 

Here C/ is a matrix from U{oo) and u ranges over the set of its eigenvalues. All but 
finitely many w's equal 1, so that the product over u is actually finite. The product 
over i is convergent, because the sum of the parameters is finite. 

Theorem 1.3. The functions x^'^\ where uj ranges over VL, are exactly the extreme 
characters of the group U{oo). 

The coordinates af, [Sf, and 7^ (or 6^) are called the Voiculescu parameters 
of the extreme character x^'^^ ■ Theorem 1.3 is similar to Thoma's theorem which 
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describes the extreme characters of the infinite symmetric group, see [Thl], [VKl], 
[Wa], [KOO]. 

The study of extreme characters of the group U{oo) was initiated by Voiculescu 
[Vo]. He proved (among other things) that all functions (1.2) are actually extreme 
characters. The fact that Voiculescu's list is exhaustive was established later by 
Boyer [Boyl] and by Vershik-Kerov [VK2]. They independently pointed out that 
Theorem 1.3 is implied by old Edrei's result [Ed] about two-sided totally positive 
sequences. 

On the other hand, Vershik and Kerov outlined in [VK2] another approach to 
Theorem 1.3, based on the idea of approximating extreme characters by normalized 
irreducible characters of the compact groups U{N). The same idea was employed 
in their previous work [VKl] on the infinite symmetric group S'(oo). 

A detailed proof of Theorem 1.3 by Vcrshik-Kerov's asymptotic method was 
given much later by Okounkov-Olshanski [OkOl] (this paper also contains more 
general results). In a special detailed proof was earlier given by Boyer 

[Boy2]. 

Here are some comments to Voiculescu's formula (1.2). 

Remark 1.4. The simplest extreme characters are of the form xiU) = det^{U), 
where A; e Z. The corresponding parameters are as follows: all of them are equal 
to zero except the first coordinates in /3+ (if /c > 0) or in P~ (if k < 0), which 
are equal to 1. 

Remark 1.5. Given a character define the function xC>?)det'^( ■ ) as the pointwise 
product x{U) det^{U)). Then x ® det'=( • ) is a character, too. If x is extreme then 
X ® det'^( • ) is extreme. In terms of Voiculescu's parameters, tensoring with det( • ) 
reduces to 

Remark 1.6. Here is a comment to the condition P^ + P^ < 1. Even if this con- 
dition is dropped, Voiculescu's formula still defines an extreme character. However, 
using the identity 

[1 + b+{u - 1)][1 + b-{u-^ - 1)] = [1 + (1 - b-)iu - 1)][1 + (1 - 6+)(^x-i - 1)], 

we can always modify the beta parameters so that the condition P'^ + Px < 1 will 
be satisfied. With this condition imposed, no freedom to change the parameters 
remains: if wi ^ u)2 then x^""^^ / X^'''^ see [OkOl, §5, Step 3]. 

Remark 1.7. Let SUipo) denote the subgroup of the matrices U e U{po) with 
determinant 1. I.e., SU{oo) is the union of the groups SU{N). Restricting any 
character to SU{oo) one gets a character of this group, and extreme characters 
remain extreme. However, it may well happen that x'^"^^^ \su (oo) equals x^^^^ \su {oo) 
for certain ui ^ ijJ2- As parameters for the characters x*''*^'' \su (oo) one may take 
a^, 7^, and a multiset S in [— |, |], which is obtained by mixing together and 
P~ . Specifically, 

B = - iW,2,... U {-P- + |b=l,2,.... 

Given a point 6 e -B, it is no longer possible to decide whether it comes from a 
coordinate of P'^ or of P~ . 
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Remark 1.8. On the set of all characters, there is a natural operation: pointwise 
conjugation. For extreme characters, it reduces to the transposition (0;+, 5+) <-> 

Remark 1.9. One can check that the functions (1.2) separate conjugacy classes 
in U (00). 

Remark 1.10. The reader might notice that each extreme character x^'^^ is a 
multiplicative function with respect to a natural product on the set of conjugacy 
classes of [/(oo): taking disjoint union of two collections of eigenvalues. Such 
a multiplicativity property holds for many other groups and can be established 
independently of the classification of extreme characters, see the survey [014]. 

2. Spherical and admissible representations 

There are two ways to establish a connection between characters and unitary 
represent ations : 

First, extreme characters of a group K parameterize its finite factor representa- 
tions, i.e., unitary representations generating finite von Neumann factors. See, e.g., 
[Th2]. 

Second, extreme characters of K parameterize irreducible spherical representa- 
tions of the pair (G, diag K) , where G = KxK and diag K is the diagonal subgroup 
in G. See [Oil] and [013, Theorem 24.5]. In these papers, it is also explained how 
to establish the relationship between both kinds of representations directly. 

In the present paper we are dealing with spherical representations. Below we 
review basic facts concerning the correspondence between characters and spherical 
representations, specialized to the particular case K = U{oo). For proofs of the 
claims stated below we refer to [013] . 

• Set G — U{oo) xU{oo) and K — diag[/(cxo). Let \& denote the set of continuous 
functions t/j on the group G that are i^-biinvariant, positive definite, and normalized 
at the unit element. Such functions will be called spherical functions. The set \E' is 
convex. 

• There exists a natural bijective correspondence x '4' between characters 
X e X{U{oo)) and spherical functions £ which is defined as follows 

X{U) = 1), i;{Uu U2) = xiUiU^'), U, C/i, C/2 e U{oo). 

• The correspondence x '0 is an isomorphism of convex sets, so that extreme 
characters exactly correspond to extreme spherical functions. 

• Any extreme function e is also extreme in a wider convex set, which is 
formed by all (not necessarily K-biinvariant) positive definite normalized functions 
on G. 

• By a spherical representation of {G,K) we mean any pair (T, ^), where T is 
a unitary representation of G and ^ is a fixed cyclic unit JC-invariant vector in the 
Hilbert space of T. The vector ^ is called the spherical vector. 

• There is a bijective correspondence V' (T, between spherical functions and 
(equivalence classes of) spherical representations, defined by 



10 



Equivalently, there is a bijection x ^ C) between characters and (equivalence 
classes of) spherical representations, defined by 

x(c/) = (T(c/,i)e,e), c/et/(oo). 

• Under this bijection, extreme spherical functions (or extreme characters) ex- 
actly correspond to irreducible spherical representations. 

• In an irreducible spherical representation, the subspace of K-invariant vectors 
has dimension 1. 

Combining these claims with Theorem 1.3 wc get the following 

Corollary 2.1. Irreducible spherical representations of the pair (G, K), where G = 
U{oo) X U{oo) and K — (XmgU{oo), are parameterized by the points u) &Vt, where 
the region VL C is defined in%l. 

An explicit realization of the irreducible spherical representations is given in 
[013], [012]. 

The class of spherical representations is part of a wider class of unitary repre- 
sentations, which is defined as follows. 



Definition 2.2. Let G, K be as above and let Kn denote the subgroup of K 
U (oo) constituted by matrices of the form ^ 



'-n 





A unitary representation T 



of the group G is called an admissible representation of (G, K) if the subspace of 
K^-invariant vectors (with varying n) is dense in the Hilbert space of T. 

It is readily proved that any spherical representation is admissible. A (con- 
jecturally, exhaustive) list of irreducible admissible representations, together with 
their explicit realization, can be found in [013], [012]. Each representation from 
this list is specified by a point a; G and some additional discrete data. It is worth 
noting that, although G is the product of two copies of C/(oo), generic irreducible 
admissible representations of (G, K) are not tensor products of two irreducible rep- 
resentations of C/(oo). 

3. The representations T^^^ 

In this section we construct a family of representations of the group G = U (oo) x 
U{oo) depending on 2 complex parameters z,w. 
Let us abbreviate 

G(N) = U(N) X U(N) c G, K(N) = diag U(N) c K, iV = 1, 2, . . . . 

We consider U{N) as the homogeneous space K{N)\G{N) with the following right 
action of G{N): 

(U, (C/i, t/2)) ^ U^^UUi , U e t/(iV), (t/i, C/2) e G{N). 

Let N > 2. Given a unitary matrix U G U{N), write it in the block form 
corresponding to the partition N = {N — 1) + 1: 

'A B' 
C D 



U 



so that D = Unn is a complex number while A is a {N — 1) x {N — 1) matrix, and 

set 

A- B{1 + D)-^C, 
A. D = -l. 



Pn{U) = 
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Lemma 3.1. The mappN defines a projection U{N) — > U{N — 1), which commutes 
with the action of G{N —1) , that is, with left and right shifts by elements ofU{N—l). 

We will call pn the canonical projection. The claim of the lemma is by no means 
new (see, e.g., [Ner3]). For the reader's convenience we give a detailed proof below. 

Proof. The fact that pat commutes with left and right shifts by elements of U (N—l) 
is evident. Next, if D = —1 then S = 0, C = 0, and A is a unitary matrix; therefore, 
in this case Pn{U) G U{N — 1). It remains to prove that if D —1 then the matrix 
V = A- B{1 + D)-^C is unitary. 

Let r C © be the graph of ^ = ^ ^ , that is 



r = {xey I a; e C^, y e y = Ux). 



-C -D 



N 



Write 



X = Xid) X2, 



y^vi® y2, 



xi, 2/1 e C 



N-l 



X2, 2/2 e . 



Intersect F with the hyperplane X2 = 2/2 and denote by Fi the image of this inter- 
section under the projection 



Clearly, Fi is described by the linear equations 



yi 




A 


B' 




Xi 






-c 


-D 




_X2_ 



x®y^xi®yi 



2/2 = X2. 



Or, equivalently, 

2/1 = Axi + Bx2 , 



2/2 = -Cxi - DX2 , y2 = X2 



Excluding X2 and j/2 from this system we conclude that Fi coincides with the graph 
ofV. 

Since U is unitary, we have {x,x) = {y,y) for any x ®y oy, equivalently, 
{xi.xi) + 1x2^ = (2/1,2/1) + l2/2p- Therefore, X2 = 2/2 implies (xi,xi) = (2/1,2/1)5 
which means that V is unitary. □ 

Remark 3.2. The above interpretation of the projection pj^ in terms of graphs 
of operators also works in the exceptional case D = —1. More generally, it can be 
used to describe the projection pm,n = Pm+i ° • • • ° Pn from U{N) to U{M) for 
any M < N . To do this we have to write the matrices U G U{N) in the block 
form corresponding to the partition N — M + (N — M) and then repeat the same 
argument. 

Remark 3.3. The projection pat is continuous on the open subset of U{N) con- 
sisting of matrices U with Unn 7^ —1 but not on the whole group U{N). Indeed, 
the first claim is evident. To demonstrate discontinuity, take N = 2 and remark 
that 

'' cos sin 



P2 



■ sm (f cos (f 



1, cos (/? 7^ — 1 
— 1, cos (fi = —1. 
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Remark 3.4. The projection Pm,n '■ U{N) — > U{M) mentioned in Remark 3.2 is 
connected with characteristic functions in the sense of Livsic et al. Write U & U (N) 
in the block form corresponding to the partition N = M + {N — M): 



U 



A B 
C D 



where A is of the size M x M. The matrix- valued function 



is called the characteristic function of U: it is an important invariant of the action 
of the subgroup 



g e U{N) I g = 



1m 
* 



U{N - M) 



on U{N) by conjugations. The function (puiC) is an inner function: ||(/>(7(C)|| < 1 
in the open disk |C| < 1 while 4>u{C) ^ U{M) on the boundary \(^\ ~ 1. See [Nerl, 
Appendix E] for further information and references to original papers. We have 
Pm,n{U) = 0?7(-l)- 

Although pn is not continuous on the whole U{N), it is clearly a Borel map. 
Hence, given a Borel measure on U{N), we may project it to ?7(A'" — 1) by means 
of pn- 

In particular, let fijy denote the normalized Haar measure on U{N). Then 
Lemma 3.1 implies that pn takes //jv to an invariant probability measure on U {N — 
1), that is, to Hn-i- 

Let S) = {D e C I |D| < 1} denote the closed unit disk. Introduce a projection 



Sat : C/(7V) ^ 2), U^D = Unn- 



This map is constant on double cosets modulo U{N — 1). Moreover, one can prove 
that En separates these cosets, i.e., if two matrices have the same (AT, A/")-entry 
then they belong to the same double coset modulo U{N — 1). 

Denote by vn the image of //jv under sn', this is a probability measure on the 
disk D. 

Lemma 3.5. 

UN{dD) = const (1 - \D\'^)^-^e{dD), 
where £ is the Lebesgue measure on 2). 

Proof. This claim is well known. It can be deduced from the fact that vn coincides 
with the image of the uniform measure on the (2A'^ — 1) -dimensional sphere e 
I ICiP + • • • ICivP = 1} under the projection C ^-^ Civ e 2)- □ 

Following [Ner3, §1.2], we combine pn and £jv into a single projection 



PN : U{N) U{N - 1) X 2). 
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Lemma 3.6. The image of /in under pn is jJiN-i x t^n- 

Proof. Indeed, if t/ e U{N) and Pn{U) = {V, D) then, for any C/i, U2 G U{N - 1), 
we have pn{U2^UUi) = {U2^VUi,D). It foUows that the pushforward of /in 
under pN splits into the direct product of fXN-i with a certain probabihty measure 
on 2). Then the latter measure must coincide with vn- □ 

Set Hn = L'^{U{N), /In) and denote by Regjy the (bi)regular representation of 
G{N) in Hn: 

{Reg^{g)f){U) = fiU2'UUi), f E Hn, UeU{N), g = {U^,U2) e G{N). 

Next, introduce the subspace C Hn of functions depending only on pjv( • )• 
Lemma 3.7. H'j^ is a G{N — 1) -invariant subspace, and we have a natural isometry 

H'^ ~ Hn-1 Vn, Vn = L\^, un). 

Proof. Indeed, this follows from the definitions and Lemma 3.6. □ 

By Lemma 3.7, any unit vector v e Vn determines an embedding KegN-i — 
Reg^, 

Hn-1 3f^f0veH%cHN. 

Fix two complex numbers z, w. Our next aim is to assign a meaning to the 
expression 

fzMNiU) = det((l + Uni + U-'r), U e UiN). 

To do this we need a preparation which concludes in Definition 3.10. 

Let Mat(iV, C) denote the space of complex NxN matrices. For X e Mat(iV, C) 
let = |(X + X*). We set 

Mat(iV, C)+ = {X e Mat(7V, C)\UX> 0}, 
Mat(7V, C)>_i = {X e Mat(7V, C)\UX> -1} = -1 + Mat(7V, C)+ . 

These are open domains in Mat(A^, C), isomorphic to a matrix wedge. 

Lemma 3.8. For any fixed z E C, the expression (1 + X)^ makes sense for 
X e Mat(A^, C)>_i and defines a holomorphic map Mat(A^, C)>_i GL{N,C). 
Moreover, 

{l + xy -{l + xy' = {1 + XY+^', X e Mat(7V,C)>_i, z,z'eC. 

Proof. Setting 1 + X = Y we must prove that Y 1-^ Y^ is a correctly defined 
holomorphic map Mat (AT, C)+ ^ GL{N, C). 

First, remark that the spectrum of any matrix Y e Mat (A?", C)+ lies in the open 
half-plane C+ = {C | > 0}. Indeed, let ( be an eigenvalue of Y and v ^ he 
any eigenvector corresponding to By the definition of Mat (AT, C)+, 

0<{{Y + Y*)v,v)^C + C, 
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which means that C e C-|.. 

Now we can define via the functional calculus: 



[ cici-Yr'dc, 

Jc 



'C 

where is correctly defined in C_|_ (we choose the branch taking value 1 at ^ = 
1 e C_|_) and C is any simple contour in C_)_ containing the spectrum of Y. 
Clearly, Y^ is nonsingular together with Y. 

Now, to check the second claim it suffices to remark that it obviously holds for 
matrices X close to 0. □ 

Corollary 3.9. For any fixed z E C, the function X i— > det((l + X)^) is correctly 
defined and holomorphic in Mat(A'", C)>_i. Moreover, 

det((l+X)^)det((l+X)^') = det((l+X)^+^'), X e Mat{N,C)>-i, z,z'eC. 
□ 

Note that, denoting by xi, . . . , xn the eigenvalues of X, we have 

N 



detiii+xr) = l[{i+xkr. 



k=l 

The right-hand side makes sense, because JJxfe > —1. On the contrary, the expres- 
sion (det(l + X))^ = (n(l + Xk))^ is ambiguous. 

Definition 3.10. Let U{N)' C U{N) denote the set of unitary matrices which 
do not have -1 as an eigenvalue. Note that U{Ny = U{N) n Mat(7V, C)>_i. By 
Corollary 3.9, the function 

fzMNiU) = det((l + Uni + U-'r) 

is well defined on U{N)'. 

Equivalently, denoting by ui,...,un the eigenvalues of U and assuming that 
none of them equals —1, we have 

N 



k=l 

On the complement of U{N)' , which is a negligible set with respect to the Haar 
measure, we agree to continue the function by 0. □ 

Thus, we have defined the function fz,w\N for any complex z, w and each = 
1,2,... . In a few lemmas below we describe some special properties of these 
functions which are used in the construction of the representations. 

Lemma 3.11. Write arbitrary N x N matrices in block form according to the 
partition N = (N — 1) + 1. The map 



X = 



A B 
C D 



Xi = A-B{1^D)-^C 
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is correctly defined in the domain Mat(A'", C)>_i and projects it onto the domain 
Mat(Ar-l,C)>_i. 

Consequently, pn maps U{Ny onto U{N — 1)'. 

Proof. First of all, note that X e Mat(A^, C)>_i implies ?R:D > -1, so that 1 + 
and the matrix Xi is well defined. 

Next, we use the argument and notation of Lemma 3.1. Consider the graph T 
A B' 



of the operator 



. In terms of F, the condition '^X > — 1 means 

X©!/ e F. 



-C -D 

{xi,xi) + |a;2p + ^{{xuvi) - X2y2) > 

When X2 = y2, this condition turns into 

{xi,xi) + U{xi,yi) > 0, a;i©yieFi, 

which means that Xi e Mat{N — 1, C)>_i. 

Together with Lemma 3.1 this implies that pn maps U{Ny to U{N — 1)'. The 
surjectivity is evident. □ 



Lemma 3.12. For any z E C and X — 



A B 
C D 



e Mat(A^, C)>_i we have 



det((i + xy) = det((i + PN{x)y) ■ (1 + Dy. 

Proof. First of all, note that all the terms of this formula make sense. Indeed, 
(1 + X)^ is correctly defined by Lemma 3.8, (1 + pn{X))^ is correctly defined by 
virtue of Lemma 3.8 and Lemma 3.11, and, finally, (1 + D)^ is well defined, because 
> -1. 

Next, we remark that these three terms are holomorphic functions in X e 
Mat(A^, C)>_i. Hence, it suffices to prove the formula when X is near zero. Then 
we may interchange the symbol of determinant and exponentiation, thus reducing 
the problem to the identity 



det(l +X)= det{l +pn{X)) ■ (1 + D), 



which is equivalent to 
det 



1 + A B 
C 1 + D 



= det((l + A) - B{1 + D)-^C) ■ (1 + D). 



This is a special case of the well-known identity for the determinant of a block 
matrix (of an arbitrary format). 



det 



a h 
c d 



det (a — hd ^c) det d. 



□ 



As a corollary we get the following formula: 

fzMN{U) = f,MN-i{PN{U)) ■ (1 + DY{1 + or 

where 

U e U{Ny, d = Unn = SNiU) e S \ {-1}. 



(3.1) 
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Lemma 3.13. The function D i— > is square integrable with respect 

to the measure on the unit disk D provided that 2^z + 2^w + TV > 0. 

Assume that this condition is satisfied and let v^^^^j^f denote the above function 
viewed as a vector of the Hilbert space Vn — L'^i'Zi, i'n)- Then we have 

2_ /■ ,n , , TT^-,2„ r(7V)r(iV + ^ + ^ + «; + «;) 



T{N + 2 + w)V{N + z + w) 

Proof. The measure vjq is given by the formula of Lemma 3.5. Using it we get 

f \{l+DY{l+D)'^\^VN{dD) = Qonsi f {1+Dy+'^ {1+Dy+'^ {l-\Df)^-^ e{dD), 
Jv Jv 

where £ denotes the Lebesgue measure. A way of calculating the latter integral 
is given in [Ner3, §1.8]. The constant is fixed by the requirement that the whole 
expression equals 1 for 2; = = 0. □ 

Lemma 3.14. Assume that + > — |. Then the function fz,w\N belongs to 
the Hilbert space Hn = L'^{U{N), /in), and 

UzmnW - nr{k + z + w)T{k + z + w) • 

Proof. Using the identification of two spaces indicated in Lemma 3.7 we can rewrite 
formula (3.1) as 

/z,u)|jv = /«,u)|Ar-i ® i^z,m|jv , N>2. 

Note that our assumption on the parameters implies that 2^z + 2^w + N > for 
any = 2, 3, . . . (even for A = 1). Consequently, V;^^^,\i^ is well defined as a vector 
of Vn. Applying Lemma 3.13 we get that fz,w\N is square integrable provided that 
fz,w\N-i is square integrable, and we have by recurrence 

||/2,«)|Ar|| — \\fz,w\l\\ ■ \\'Vz,w\2\\ ■ ■ ■ \\Vz,w\n\\- 

It remains to check that the function fz,w\i is square integrable on U{1) (the 
unit circle) provided that ^z + > — | , and 



11/ 



,|2 _ T{l)T(l + z + z + w + w) 
zMiW - r{l + z + w)T{l + z + w) ■ 



That is, denoting by \du\ the Lebesgue measure on the unit circle \u\ = 1, 



Ji^uY-'ii^^r- r(i + . + ^)r(i + . + .)- 

This can be proved by the same argument as in [Ner3, §1.8]. 

Note also that the value of the above integral coincides with the value of the 
expression of Lemma 3.13 formally specialized at A = 1. This coincidence has an 
explanation. Indeed, consider the expression for the measure ujy on the disk D 
given in Lemma 3.5, and assume that the parameter A is a real number. Then, 
as A tends to 1 from above, the measure degenerates to the uniform measure 
concentrated on the unit circle. □ 



\du\ T{l)T{l + z + z + w + w) 



The above results make it possible to give the following definition. 
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Definition 3.15. 

• Let us fix arbitrary z, w G C. If is large enough {N > —2^z — 2^w) then 
the vector v^,u,|jv G Vn is well defined, and we can normalize it by setting 



II ^z,w\N\\ 

where the norm is given by Lemma 3.13. 

• Define an isometric embedding L^^^^j^ : i^jv-i Hn using the identification 
of Lemma 3.7: 

Hn-1 9/^/0 <,„|iv e Hn_i <^Vnc^H%gHn. (3.2) 

It commutes with the action of G{N — 1). Consequently, we can form the inductive 
limit Hilbert space H — lin^ Hm and the natural unitary representation hn^Regjv 
in H. We denote it by Tg,^. 

• If + > — I then the functions fz,w\N cire square integrable and we can 
normalize them: 

„/ _ fz,w\N 

J Z.wlN Mr 11 ■ 

Then (3.2) implies existence of the inductive limit vector 

It is i^-invariant, because all the functions fz,w\N are constant on conjugacy classes. 

• It is worth noting that the definition of this distinguished vector makes sense 
only when 'Rz + 'Rw > — | • 

Remark 3.16. Removing the normalization of the function V2,u,|jv we get a map 
Lz,w\Ni which differs from L^^^^j^j- by a scalar multiple and has the form 

{LzMNfm = f{PN{U))vz,^iN{UNN), U e U{N). 

It is worth noting that Lz^w\N makes sense for any A^, irrespective of the values of 
the parameters z, and can be applied to any function / on U{N — 1). By virtue 
of (3.1), we have 

Lz,w\^ • /^,w|JV-l ^ fz,w\N ■ 

□ 

The formula (3.2) describes the embedding Lz^w\n '■ H^-i Hn. More gener- 
ally, for M < (where M is large enough), we shall now describe the embedding 

Lz,w\M,N '■ Hm Hn, Lz^w\m,n = Lz^w\n o ■ ■ ■ o Lz^w\m+i ■ (3.3) 



Write any U e U{N) in block form 



according to the partition N 



A B 
C D 

M + [N — M). Then D lies in the matrix ball 2) n-m, the set of complex matrices 
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of size (N — M) x (N — M), with norm < 1. Generalizing the definitions of pn and 
En, consider the maps 

eM,N ■■ U{N) -> Dtv-m , U^D, 
Pm,n :U{N)^U (M) X Dn-m , Pm,n Pm.n x £m,n , 

where Pm,n was introduced in Remark 3.2 (if U e U{Ny then Pm,n{U) — A — 
B{1 + D)~^C). Let iy'M,N be the image of the Haar measure under the map 
£m,n '■ U{N) Dn-m, and let Vm,n denote the Hilbert space L'^{Dn-m, i^m,n)- 
Next, let C Hn = LF'{U{N), iin) be the subspace of functions depending on 

Pm,n{ ■ ) only. As in Lemma 3.7, we have a natural isometry between H^ j^ and 
Hm ® Vm,n] we use it to identify these two spaces. 

Set = Dn-m H Mat(A^ — M)>_i; this subset of Dn-m contains the 

interior of the matrix ball. Generalizing the definition of the vector v^^wlN (see 
Lemma 3.13), we introduce a function Vg^^\M,N on Dn-m as follows: 

r det((l + L»ndet((l+:Dn, DeD'^_^; 

VzMmM^) = \ n^T> \T>' ^^^ ^ 

L u, n e x)n-m \ ^n-m • 

Proposition 3.17. Let M be so large that the embeddings Lz^w\n o,re well defined 
for all N > M. Fix N > M. 

(i) The function Vg^^\M,N defined in (3.4) lies in the Hilbert space Vm,n = 
L'^iDM,N, ^m,n)- 

(ii) The embedding L;^^^\m,n defined in (3.3) maps Hm into the subspace -ff^f at 
of Hn- Under the identification H^ ^ — Hm ® Vm,n , this embedding has the form 

T f f ^ ' I — '^g.wlM.AT 

\\^z,w\M,N\\ 

Proof. From the definition (3.3) it follows that 

{LzMNf){U) = f{PMAU))9{U), (3.5) 

where ^ is a certain function not depending on /. We shall prove that g{U) is 
proportional to Vz^y^\M,N{^M,N{U)). It will follow that the image of Lz^w\N fies in 

w\m n then all the claims of the proposition will become clear. 

It is convenient to pass to the maps defined in Remark 3.16. Then we get, 
similarly to (3.3), a map Lg^^\M,N, which differs from Lz^nj\M,N by a scalar multiple. 
Similarly to (3.5), we have 

(£.,^|M,Ar/)(t^) = f{pM,N{U)mU): (3.6) 

where ^ is proportional to g. According to Remark 3.16, we may apply formula 
(3.6) to any function /, not necessarily a square integrable one. So, we may take 

/ = fz,w\M- By the last claim of Remark 3.16, Lz^u}\n takes fz,w\M to fz,w\Ni which 
implies 

9[U) = f 7^ . (3.6) 

Jz,w\m[Pm,n[U )) 

On the other hand, the same argument as in the proof of Lemma 3.12 shows that 
the right-hand side is equal to f z,tu|M,7v(-D), where D — eM,N{U) G Dn-m- 

Thus, g{U) = Vz^^\m,n{D)- Since g is proportional to g, this concludes the 
proof. □ 
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Theorem 3.18. All representations constructed in Definition 3.15 are admissible 
in the sense of Definition 2.2. 

Proof. Assume as above that M is a sufficiently large natural number, so that the 
embedding Hm <Z H is well defined. By the very definition, the subspace UmHm 
is dense in H. 

On the other hand, denote by H'j^ the subspace in H formed by all i^M-invariant 
vectors (recall that Km C K ~ U{oo) is the subgroup of matrices of the form 

^ , see Definition 2.2). The subspaces form an ascending chain (because 

the subgroups Km form a descending chain). We must prove that Um-H^m dense 
in H. To do this we shall prove that H'j^ D Hm- 

For any N > M, let Km{N) denote the subgroup in K ~ U{oo) that is the 
intersection of Km with U{N). Clearly, Km{N) is isomorphic to U{N — M) and 
we have Km = ^n>mKm{N). Note that the function Vz^w\m,n on the matrix ball 
D N-M is invariant with respect to conjugation by unitary matrices from U{N — M). 
Note also that, for any U G U{N), the matrix Pm,n{U) does not change when 
we conjugate U hy a matrix from Km{N) C U{N). Combining this with the 
description of the embedding Hm H^ we conclude that the vectors from Hm 
are invariant with respect to Km{N). Since this holds for any N, it follows that 
all these vectors are KM-invariant, i.e., Hm C H'^. □ 

4. The space iX of virtual unitary matrices and 

ANOTHER DESCRIPTION OF THE REPRESENTATIONS T^^w 

The construction of the representations Tzw described in §3 is quite similar to 
the second construction of the "generalized regular representations" in [KOV]. 
In this section we present the counterpart of the first construction from [KOV]. 

Definition 4.1. Let il = \im .U(N) be the projective limit of the spaces U{N), as 
N ^ oo, taken with respect to the projections Pn '-U {N) — > U {N — 1). By analogy 
with the space of virtual permutations from [KOV] we call H the space of virtual 
unitary matrices. 

By definition, a point of H is an arbitrary sequence x = (xn), where xn & U (N) 
for any = 1, 2, . . . , and Pn{xn) — xn-i for any N > 2. We denote by ttjv the 
natural projection map IX U{N) sending x to xn. 

There is a natural embedding U{oo) > il assigning to a matrix U G t/(oo) a 
sequence x = (xn) such that xn = U provided that N is so large that U G U{N) C 
U (oo). Thus, the image of U (oo) in il consists of the stabilizing sequences x = (xn)- 

However, general elements of the space il cannot be interpreted as unitary ma- 
trices. 

Recall that the map pn is continuous on the open subset U{Ny C U{N) but 
discontinuous on the whole space U (N) (Remark 3.3). This is an obstacle to equip- 
ping the space il with a natural topology. However, pN certainly are Borel maps, so 
that il has a natural Borel structure. We keep in mind this structure while speaking 
of measures on il. 

Definition 4.2. By Lemma 3.11, pn maps U{Ny onto U{N — 1)', which makes it 
possible to define the following subset in il: 

il' = iimf/(iV)' Cil. 
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Definition 4.3. A subset of il will be called negligible if for any N, its image under 
TTjv : It — > U{N) is a null set with respect to the Haar measure on U{N). 

For instance, is a negligible set. Let us agree to identify functions on ii 

that coincide outside a negligible set. Let also agree that a function is well defined 
on il if it is defined outside a negligible set. 

Definition 4.4. We define a right action {x,g) i— > x.g of the group G = U{oo) x 
U (oo) on the space It by 

7TN{x.g) = U^^ttn{x)Ui, X e il, ge {Ui, U2) e G, 

where N is so large that g e G{N) = U{N) x U{N). The correctness of this 
definition follows from the basic equivariance property of the canonical projection 
Pn, see Lemma 3.1. 

In other words, this action arises from the right actions of the groups G{N) 
on the spaces U{N) defined in the beginning of §3. On U{oo) C il, this action 
coincides with (t/, (t/i, U2)) U2^UUi. Note also that the action of the subgroup 
K = diagC/(cxD) arises from the actions of the groups U{N) on themselves by 
conjugations. 

Note that the shift of a negligible set by an element of G is negligible, too. 

Definition 4.5. A function on il is called cylindrical if it has the form F{x) = 
Fn{7^n{x)) for a certain N and a certain function Fn on U{N). More generally, we 
extend (in an evident way) this definition to functions defined outside a negligible 
set. 

Lemma 4.6. Let z,w & C, g & G, and a; e il' fl (il'.^r"^). If N is large enough 
then the expression 

fz,w\N{'^N{x.g)) 
fz,w\N{^N{x)) 

does not depend on N. 

Proof. Indeed, assume g G G{N — 1), and let us prove that the expression above 
does not change when is replaced by A — 1. Recall that the function fz,w\N is the 
product of two determinants (which correspond to the particular cases w — and 
2; = 0, respectively). We shall check the above claim for the first determinant; for 
the second determinant it is proved similarly, so that the desired claim will follow. 

Let X = 7Tn{x) and g = {Ui, U2); then n]y{x.g) — U2^XUi. Since g G G(A — 1), 
the matrices Ui, U2 lie in U{N - 1). It follows that pn{U2^XUi) = U2^pn{X)Ui 
(Lemma 3.1) and that the matrix entry D = Ajvjv does not change when X is 
replaced by Ui^^XUi. 

Then, applying Lemma 3.12, we get that the ratio 

det((l + U^^XUiY) 
det((l+A)^) 

does not change when X is replaced by pn{X), which concludes the proof. □ 
As a corollary we get 
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Proposition 4.7. Fix arbitrary z,w e C For g E G and a; e It, set 

Cz w{x, g) = the stable value of ^^M^^ ^)) as A?" — > oo. 

For any g, this is a correctly defined cylindrical function in x in the sense of Defi- 
nition 4-5. Furthermore, Cz^wi^^g) possesses the multiplier property 

Cz,w{x,gi)Cz,w{x.gi,g2) = Cz,^{x,gig2), x e ii, gi,g2eG. 

Finally, 

Cz,n,{- ,9) = '^ ioi geK. 

Proof. The first claim follows from Lemma 4.6. The second claim is evident. The 
third claim follows from the fact that the function fz,w\N on U{N) is central (con- 
stant on conjugacy classes). □ 

Fix an arbitrary s e C with 3?s > — |. For each N = 1,2, .. . we consider a 
measure /xj^^ on U (N) such that 

where, as above, /j^r is the normalized Haar measure on U{N). When s = 0, this 
measure reduces to //jv- Further, using the formula of Corollary 3.9 and Lemma 
3.14, we get that 



l^N{dU) \\fz,w\N\ 



2 



This implies that n)^ is a probability measure. Since fz,w\N is central, iij^ is 
invariant under the action of U{N) on itself by conjugations. 

Lemma 4.8. The family {Atjv^}jv>i is consistent with the projections pn, i-e., the 
pushforward of ji^^ under pn is A*jv-i- 

Proof. Same reasoning, based on Lemma 3.12, as in the proof of Lemma 3.13. □ 

This makes it possible to give the following 

Definition 4.9. For any s e C, 3?s > —1/2, we denote by ji^^^ the projective limit 
of the family {jj!"^} as A?" — > 00. This is a probability Borel measure on il. 

Note the following properties of the measures /x^*) : 

• ji^^^ is invariant with respect to the action of K on il; 

• the measure (jP is the projective limit of the Haar measures //jv; consequently, 
it is G-invariant; 

• negligible sets are null sets with respect to any measure 
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Proposition 4.10. For any s e C with 3?s > — |, the measure fj,^^^ is G -quasiinvariant. 
Its Radon-Nikodym derivative 

[i^^i\dx) 

is a cylindrical function in x. Moreover, for any complex z, w such that z + id = s 
we have 

Ts{x,g) = \C:,,^{x,g)\'^, a; Gil, g E G, 
where Gz,w{x^g) is the multiplier introduced in Proposition 4-7. 

Proof. Indeed, this follows from the definition of the measures fi^^^ and Proposition 
4.7. □ 

Theorem 4.11. Let z^w E C be such that > — |, z+w = s, and let Gz,w{x^ g) 
he the multiplier introduced in Proposition ^.l. The following formula defines a 
unitary representation ^ of the group G = U{oo) x U{oo) in the Hilbert space 

{T^,Mf){x) = f{x.g)Cz,w{x,g), g E G, f E H, x E ii. 
Further, the constant function 1 E L^(il, /x'^'^^) is a unit K-invariant vector in 

H. 

We call 1 the distinguished vector of the representation ^. 

Proof. The correctness of the definition follows from Proposition 4.10, and the 
second claim follows from the JC-invariance property of the multiplier, see the last 
claim of Proposition 4.7. □ 

Proposition 4.12. Let z,w E C and s = z + id. Assume that 5Rs > — |. There 
exists a Hilbert space isometry between the space of the representation T^^^ 
and the space H of the representation Tz^w constructed in section 3; this isometry 
intertwines the representations Tlz,w ^'^^ takes the distinguished vector 

1 E to the distinguished vector ^z,w E H introduced in Definition 3.15. 

Proof. RecaU that H = lin^ifiv, where if at = L'^{U{N) t/jn)- On the other hand, 
since fi^^^ = ^m/xj^-*, we have = lin^iJ where I^j^ = L'^{U{N), ^^■*). For each 
N = 1,2, . . . , the operator of multiplication by the function 

J., ( \ — -f^M^i • ) 

/ z,w\N\ ' ) \\ f II 

defines an isometry I^jq — > Hn taking the constant function 1 to the vector f'z 
By the very construction, these isometries are consistent with the embeddings 

H M ILn+1 Hjsf — > -ffjv+i and, consequently, define an isometry H ^ H. 
Clearly, the isometry I^—^H takes 1 to ^z,w Moreover, again by the construction 
of Theorem 4.11, ^ — > ii is an intertwining operator between the representations 
T,^„andT,,„. □ 

Remark 4.13. The above results can be extended, with appropriate modifications, 
to the case when z,w are arbitrary complex numbers (and s = z + id, as usual). 
When 3fis < -i, the measure n^^^ is infinite and the constant function 1 is not a 
vector of I£. Recall that the definition of the distinguished vector ^z,w also fails 
in this case. The equivalence of Tz^w and Tz,w remains valid for all values of the 
parameters. 
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5. Measures on the space of Hermitian matrices 
Here we give another description of the measures /x^*^ which were introduced in 

§4. 

Let Herm(A'") denote the space oi N x N complex Hermitian matrices. 

Definition 5.1. We introduce a bijection U ^ X between U{N)' C U{N) and 
Herm(A'") as follows: 

where i = is identified with the scalar matrix i-In- We call U{N)' Herm(A'') 
the Cayley transform and Herm(iV) — > U{N)' the inverse Cayley transform. 

We define the projection p'j^ : Herm(A'') — > Herm(A'^ — 1) as the operation of 
removing the iVth row and the iVth column from a N x N matrix. 

Lemma 5.2. The following diagram is commutative 

U{Ny Herm(7V) 



Pn 



U{N - ly Herm(iV - 1) 

Proof. Indeed, this is verified by a direct calculation. The easiest way is to use 
the interpretation of pn in terms of graphs of operators, see the proof of Lemma 
3.2. □ 

Proposition 5.3. The Cayley transform takes the Haar measure jiN on U {Ny to 

the measure 

ruNidX) = const det(l + XX*)'^ e{dX) 

on Herm( AT) . Here £ denotes the Lebesgue measure. 

More generally, for any s e C with 3?s > — |, the measure //j^-* is transformed 
to the measure 

m%\dX) = const det((l - iX)-') det((l + iX)'^) det(l + XX*)'^ £{dX). 

Proof. Direct computation. □ 

Note that for real values of s the latter expression can be simplified: 

m%\dX) = const det(l + X^)"^"^ s e M. 

Corollary 5.4. Fix s e C, Us > -\. The measures rri^^ with varying N are 
consistent with the projections p'^. 

Proof. Indeed, this follows from Lemma 5.2, Proposition 5.3, and the similar claim 
for the measures //^'*. □ 

Of course, a direct verification of Corollary 5.4 is also possible. For real values of 
s it was first carried out by Hua, see the proof of Theorem 2.1.5 in his remarkable 
book [Hua]. 

The considerations above lead to the following 
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Definition 5.5. Let Sj denote the space of all infinite Hermitian matrices. We may 
view it as a projective limit space: S) = ^im Herm(A^). For any s e C with 3?s > — ^, 

the family {m^^^}N=i,2,... defines a probability measure on the space Sj, which will 
be denoted by m^^^ and called the Hua-Pickrell measure (with parameter s). 

The Hua-Pickrell measures are studied in detail in [B03]. The measures with 
real parameter s are exact counterparts of measures introduced by Pickrell [Pi] 
(instead of i^, he considered the space of all infinite complex matrices). 

The group U{oo) operates on the space by conjugations, and all the measures 
m*-*^ are clearly invariant with respect to this action. 

The Cayley transforms U{N)' Herm(Ar) (Definition 5.1) define a bijection 
il' S), which is an isomorphism of measures spaces (il', /U*^^-*) {Sj^m^^'^) for 
any s. Since il \ il' is a negligible set, this can also be viewed as an isomorphism 
(il,/i(")) ^ (53, m(")). 

Remark 5.6. The bijection il' ^ takes the action of G on il to an action by 
fractional-linear transformations on Sj. The transformation of a matrix X E S) hy 
an element g = {Ui, U2) £ G has the form 

X ^ {Xh + d)-^{Xa + c), 



Given g G G, this transformation is defined almost everywhere. 

Remark 5.7. The above results can be extended to the case of an arbitrary com- 
plex parameter s, cf. Remark 4.13. However, when 3?s < —1/2, the measures 
become infinite. 

6. The characters Xz,w 

Let X be a character of U{oo). Then, for each = 1, 2, . . . , the restriction of x 
to U{N) C U{oo) is a character of U {N) (in the sense of Definition 1.1). According 
to Example 1.2, we have 

x\u(N)= Yl Pn{X)x\ (6.1) 

xeu{Nr 

where PAr(A) are nonnegative coefficients whose sum is equal to 1, and x^ are the 
normalized irreducible characters ofU{N), see (1.1). Note that the series converges 
uniformly on U{N), because |x'^( • )l — 1- 

Any character is uniquely determined by the collection of its coefficients Pn{\). 
Indeed, the coefficients with fixed N determine the restriction of the character to 
U{N), and if we know these restrictions for all N then we know the character itself. 

Definition 6.1. Following the general scheme described in §2, we introduce a 
family of characters attached to the representations Tg^yj , as follows. Let z,w E <C 
satisfy the condition ?R.z + > — | ensuring the existence of the distinguished 
K-invariant vector ^^^yj in the representation T^^yj, see Definition 3.15. We consider 
the matrix coefficient determined by this vector and then pass to the corresponding 
character, which we denote by Xz,w' 



a b 




c d 





U1+U2 _A U1-U2 

2 2 



U1-U2 



U1+U2 
9. 
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Lemma 6.2. Assume U e U (N) G U (oo) . Then 

Xz,w{U) = .. ^ no / fz,w\N{VU)fg^^lN{V)fXN{dV). 
||j2,w;|Ar|| JuiN) 

Proof. By the very construction of the representations T^^w, if f/ G U{N) then 

Jt'(n) ' 

which is equivalent to the desired formula. □ 

Our aim is to describe explicitly the expansion (6.1) of the characters Xz,w 
We shall interpret the labels A G U{N)"" of irreducible characters of U{N) as 
signatures of length N, i.e., as ordered A^-tuples of integers: 

A= (Ai > ••• > Aiv), AiGZ. 

Theorem 6.3. Assume + 3f?to > — |. Denote by Pz,w\n{^) the coefficients in 
the expansion (6.1) of the characters Xz,w We have 

N ^ 2 



PN{X\z,w)^{SNiz,w))-^-l[ 



i=l 



T{z -Xi + i)T{w + N+l + Xi-i) 



(6.2) 

where 

^ ( \ ^Y] T{z + z + w + w + i) 

and Dimjv(A) is the dimension of the irreducible character x^j given by Weyl's 
formula: 

Dim^(A)= n 

The proof of Theorem 6.3 is partitioned into a few lemmas. 

Note that the irreducible characters x^, where A ranges over U{N)^, form an 
orthonormal basis in the subspace of Hn — L'^{U{N), hn) constituted by central 
functions. Therefore, there exists an expansion 

fz,w\N = ^ Cz,w\NX'^ 

xeu{Nr 

with certain coefficients Cz^w\N- 
Lemma 6.4. We have 

1^ |2 
\'^z,w\N\ 



Pn{X \ z,w) = 



l|/2,w|iv|P 



Proof. This follows from Lemma 6.2 and the formula 

/ x\VU)^^MdV) = 6^,x\U), X,f,eGTN, 

JU{N) 

which in turn follows from Schur's orthogonality relations. □ 

Since the norm II/^^wIatH is known (Lemma 3.14), our problem is entirely reduced 
to evaluating the coefficients Cz^w\n- 
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Lemma 6.5. Assume that 

oo 

f,iu)= J2 c(0«'' c(OeC, 

l=—oo 

is a function on the unit circle \u\ = 1. For any N — 1,2, . . . consider the following 
function on U{N), which is constant on conjugacy classes: 

fN{u) = fiM...f,M, 

where U ranges over U{N) and {ui, . . . ,uisr) denotes the spectrum of U . Expand 
fjsf in the "Fourier series" on the irreducible characters, 

fN= 

Then the "Fourier coefficients" c(A) are given by determinants of order N, 

c(A) = det[c(Ai - i + j)]i<i,j<N- 

Proof. This is a well-known combinatorial fact, based on Weyl's character formula. 
See, e.g., [Vo, Lemme 2] or [Hua, Theorem 1.2.1]. □ 

Observe that, by Definition 3.10, 

/2,«;|iv(diag(wi, . . . , un)) = • • • /«,«,|i(wiv), 

as in Lemma 6.5. By virtue of this lemma, the problem of evaluating the "Fourier 
coefficients" of fz,w\N is split into two parts: first, calculate the Fourier coefficients 
of fz,w\i and, second, calculate the above determinants. 

Lemma 6.6. The coefficients of the Fourier expansion 

oo 
— oo 

have the form 



_ T{l + Z + W) 

c^Mi^i) r{l + z-l)T{l + w + l)- 
Proof. Setting u = e*^ we have 

c.Hi(0 = ^ f {i + uY{i + uru-Ue. 

To evaluate this integral one can, e.g., reduce it to [Er, 1.5 (30)]. See also [Ner3]. □ 
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Lemma 6.7. The determinants of Lemma 6.5 corresponding to the Fourier coef- 
ficients of Lemma 6. 6 can he explicitly calculated: 



N 



/w TT T(z + w + i)T(i) 

CM^W = n r(. - A. + inn, + N + l + X,-i) - C-^* 

Proof. We will reduce the determinant in question to a known one, given in [Kra, 
Lemma 3]: 



det 



-, N 

i-1 N-l 

Ylixi + Ofc) • Y[ {xi + bk) 

k=l k=j 



n i^i-^j)- n K-^i)- 

^<i<3<N ^<i<j<N-l 



J i,j=l 

(6.5) 

To do this we abbreviate li — — i and we transform our determinant as follows: 
(^^ (r(l + ^ + «;))^ , ... 

= nLr(.-A. + .)r(.^^ + i^A.-.) ■ 

where 

V{z-li)V{w + N + l + 



V{l + z-li-3)nw+l + li+3) 

= {z-h-\:)...{z-h-j + V)- {w + N + h)... {w + h+j + l). 

V ' ^ V ' 

j-1 N-j 

Then the reduction to (6.5) is carried out by setting 

Xi = k, Gj = -Z+j, bj =w + l+ j. 

Then, after simple transformations, we get (6.4). □ 

Proof of Theorem 6.3. Follows from the lemmas above and the formula for the 
norm in Lemma 3.14. □ 

Remark 6.8. Note some symmetry properties of the characters Xz,w- 
• Conjugation: 

XzAU)^Xu,AU), ueu{oc). 

Indeed, this follows from Lemma 6.2 and the fact that 



fzMNiU) = fy,,,\N{U), UeU{N). 

• The character Xz,w is invariant under the symmetries of the parameters gen- 
erated hj z ^ z and w ^ w. Indeed, this follows at once from the corresponding 
symmetry of the coefficients P/v(A | z,w). Note that this fact is not obvious from 
the construction of the characters Xz,w 

• Let A* = (— Ajv, . . . , — Ai) denote the dual signature to A: this is the label of the 
conjugate irreducible character x'^- Another symmetry property of the coefficients 
Pjv(A I z,w) is as follows: 

P/v(A \ z,w) = Piv(A* I w,z). 

Theorem 6.3 is an analog of Proposition 4.8 in [Pi]. This theorem leads to an 
important consequence for the representations T^^^,, cf. Lemma 4.5 in [Pi]. 
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Corollary 6.9. Let z,w E C be such that + > —1/2 and z,w ^ Z. Then 
the distinguished vector ^z,w (see Definition 3.15) is cyclic. 

Proof. Indeed, for nonintegral values of z,w, the coefficients Piv(A | z,w) are non- 
vanishing for all N and all A G U{N)^. This implies that the vector fz,w\N ^ Hjy is 
a cyclic vector of the representation Reg^ for any N. This concludes the proof. □ 

Remark 6.10. Another possible way of evaluating the coefficients Cz,w\n is to use 
Neretin's results, see [Ner4]. 

7. Other series of characters 
For two signatures v and A, of length N — 1 and N, respectively, write u ^ X ii 
Ai > z^i > A2 > > ■ ■ ■ > ^N-i > Aat . 

The relation u ~< X appears in the Gelfand-Tsetlin branching rule for the irreducible 
characters of the unitary groups, see, e.g., [Zh]: 

\u{N-i)= ^ X'' • 

v: v^X 

Definition 7.1. The Gelfand-Tsetlin graph GT is a -graded graph whose ATth 
level GTjv consists of signatures of length N. Two vertices u e GTjv-i and A e 
GTn are connected by an edge if u -< X. We agree that GTq consists of a single 
element denoted as 0; it is connected to all A G GTi. □ 

Definition 7.2. For u G GTn-i and A G GTn, where iV = 1, 2, . . . , set 

{ Dimjv_i ly jy ^ x 
Dimjv A ' ' 
0, z/ 7^ A. 

This is the cotransition probability function of the Gelfand-Tsetlin graph; it satisfies 
the relation 

J2 g(z^,A) = l, VAgGTat. 

We agree that q{0, A) = 1 for all A G GTi. □ 

One can imagine that each edge (i/, /i) of the graph GT is equipped with a label, 
which is the positive number q{i', A). Note that the Gelfand-Tsetlin branching rule 
is equivalent to the relation 

X^\uiN-i)= Yl (7-1) 

i/eGTjv-1 

Definition 7.3. Assume that for each N = 0,1, . . . we are given a probability 
measure Pjv on the discrete set GTjv. The family {Pn} is called a coherent system 
if 

Pjv-iH= Yl Qi^:>^)PN{X), AT =1,2,..., ueGTn-i. (7.2) 

□ 

Note that if Pn is an arbitrary probability measure on GTn then (7.2) defines a 
probability measure on GTjv-i (indeed, this follows at once from the above relation 
for g(zy. A)). Thus, in a coherent system {Pn}, the A"th term is a refinement of the 
(A^ - l)th one. 
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Proposition 7.4. There exists a natural bijective correspondence x {Pn} be- 
tween characters of the group U (oo) and coherent systems on the Gelfand-Tsetlin 
graph. The correspondence is defined by the relations 

XN= Pn{X)x\ XN-.^xluiN), iV=l,2,.... (7.3) 



Proof. Let % be a character of U{oo). We repeat the argument at the beginning of 
§6. Since xn is a character of U{N) in the sense of Definition 1.1 (N = 1,2,...), 
we get, according to Example 1.2, the expansion (7.3), where P7v( • ) is a probabihty 
measure on GTn- Then the relation (7.2) follows from (7.1) and the evident relation 
Xn \u{n-i)= Xn-1- 

Conversely, let {Pn} be a coherent system. We define by means of (7.3) a 
sequence {xn} of characters of the groups U{N). The coherency property (7.2) 
ensures that xn \u{n-i) = Xn-i for any = 2, 3, . . . , so that there exists a function 
X onU (oo) such that x \u{n)= Xn for all N = 1,2, ... . Obviously, x is a character 
oft/(oo). □ 

Using Proposition 7.4 we will construct new series of characters by analytic 
continuation of the formulas (6.2)-(6.3). 

Let z, z', w, w' be complex parameters. For any N = 1,2,... and any A e GTn 

set 

PJV(A I z, z'w, w') = Dim^(A) 

N 

X 



n 



\ T{z - Ai + i)T{z' -Xi + i)Tiw + N + 1 + Xi- i)Tiw' + N + 1 + - i) ' 

where DimAr A was defined in §6. Clearly, for any fixed and A, -P^(A | z, z', w, w') 
is an entire function on C"^. This expression was obtained by analytic continuation 
from the expression given in (6.2). 
Set 

V = {{z, z', w, w') eC^ \?li{z + z' + W + w') > -1}. 
This is a domain (a half-space) in C^. 

Proposition 7.5. Fix an arbitrary N — 1,2, . . . . The series of entire functions 

J2 Pn{^\z,z',w,w') (7.4) 

AgGTjv 

converges in the domain T>, uniformly on compact sets. Its sum is equal to 
c / / '^ = n T{z + z' + w + w' + i) 

cyN{z, z,w,w ) II r{z + w + i)r{z + w' + i)r{z' + W + i)r{z' + w' + i)r{i) 

(7.5) 

Proof. Let us prove that for any A e GT^v 

N 

\P'j^{X I z, z, w, w')\ < const JJ(1 + \x.\)-^i^+^'+^+^'+'^) , (7.6) 

i=l 
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uniformly on compact sets in V. 

Indeed, using the formula (r(2;)r(l — z))~^ = sin(7r2;)/7r we get 



1 

T{z -X^ + i)T{z' -X^ + i)T{w + N + 1 + X^- i)r{w' + N+l + X,-i) 
_ sin(7r2) sm{nz') T{-z + 1 + Aj - i)T(-z' + 1 + Xi - i) 
~ TT^ r{w + N + l + Xi- i)T{w' + N + l + Xi-i) 

_ sin(7r?(') sin(7r?(;') T{-u: - N - X, + i)T{-ti:' - N - X, + i) 

~ ^ r{z -Xi + i)T{z' -Xi + i) ■ 

Suppose I All — > oo. Then, applying the first or the second equality above (depending 
on whether Aj » or A^ ^ 0) and the asymptotic formula 

lim ^^^=^'^-', a,beC, 

x^+oo L [0 + X) 

we get 



T{z -Xi + i)T{z' -Xi + i)T{w + iV + 1 + Ai - i)T{w' + N + 1 + Xi - i) 

< const (1 + |;^.|)-SR(z+.'+«,+»'+2iV) ^ ^j ^^ 

where the estimate is uniform on compact sets in V. 

Next, Weyl's formula implies that Dim^ A is a polynomial in Ai, . . . , Ajv of degree 
— 2 with respect to each variable. Combining this fact with (7.7) we get (7.6). 

The bound (7.6) ensures the convergence of the series (7.4), uniformly on com- 
pact sets in V, to a holomorphic function in V. It remains to show that this function 
coincides with (7.5). To do this we remark that (7.5) is a holomorphic function co- 
inciding with the function Sn{z, w) (see (6.3)) on the subset z' = z, w' = w in T>. 
For this subset, the claim holds by virtue of Theorem 6.3. Clearly, this subset is a 
set of uniqueness for holomorphic functions in T>. This implies that the claim holds 
on the whole domain V. □ 

Note that in the special case N = 1, the set GTi is simply Z and the identity 

P[{X \ z, z,w,w') — Si{z, z,w,w') 

AeGTi 

is equivalent to the well-known Dougall's formula, see [AAR, Chapter 2, Theorem 
2.8.2 and Exercise 42(b)], [Er, §1.4]). 

Definition 7.6. The set of admissible values of the parameters z,z,w,w' is the 

subset X>adm of the quadruples {z, z,w,w') e V such that: 
First, P'jsfiX I z, z, w, w') > for any N and any A e GTn- 
Second, for any A^, the above inequality is strict at least for two different A's. 
A quadruple {z, z, w, w') will be called admissible if it belongs to I'adm- 

Consider the subdomain 

Vo = {{z, z', w,w') eV \ z + w, z + w', z' + w,z' + w'j^ -1, -2, . . . } 

= {{z, z, w, w') e V I Sn{z, z' , w, w') ^ 0}. 
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For any (z, z, w, w') e Vq we set 



Pjv(A I z, z,w,w') 



P'^jX I z,z,w,w') 
Sn{z, z, w, w') 



iV=l,2,..., AeGT 



(7.8) 



Note that Padm C Vq (indeed, if [z, z,w,w') G X'adm then Sn{z, z,w,w') is 
strictly positive), so that formula (7.8) makes sense for any admissible {z, z,w,w'). 

Proposition 7.7. For any {z, z,w,w') e Vq, the expressions (7.8) satisfy the co- 
herency relation (7.2). 

Proof. Plug in P/v_i(z/) = Pn-i{u \ z,z,w,w') and Pn{X) — Pn'(A | z,z,w,w') to 
(7.2). First of all, since < g(z^. A) < 1, the series in the right-hand side of (7.2) 
converges, uniformly on compact sets in Vq, by virtue of Proposition 7.5. Then we 
apply the same argument as in the proof of Proposition 7.5. Namely, we remark 
that the relation holds provided that z' — z, w' — w, and then conclude that it 
must hold for any {z, z, w, w') e Vq by analytic continuation. □ 

Corollary 7.8. For any (z, z, w, w') e I^adm? the expressions (7.8) form a coherent 
system. Hence, by Proposition 7.4, there exists a character Xz,z,w,w' corresponding 
to this coherent system. 

Our aim is to describe the set Vadm explicitly. 
Define the subset Z C as follows: 



where "princ", "compl", and "degen" are abbreviations for "principal", "comple- 
mentary" , and "degenerate" , respectively. The terminology is justified by the fol- 
lowing lemma. 

Lemma 7.9. Let {z, z') G C^. 

(i) The expression (r{z — k + l)T{z' — k + 1))~^ is nonnegative for all k E Z if 
and only if {z, z') G Z. 

(ii) // {z, z') G .Zprinc U 2^compi then this expression is strictly positive for all 



(iii) If {z, z') G 2^degen,m then this expression vanishes for A; = m + l,m + 2, ... 
and is strictly positive for k = m,m — 1, . . . . 

Proof. Assume first that both z and z' are nonintegral. Then the expression (r(^ — 
k + l)r{z' — k + l))~^ does not vanish for any k G 'Z. Clearly, it is strictly positive for 
all /c G Z whenever (z, z') is in ^princ or in ^compi- Let us check the inverse claim. 
Dividing T{z-k + l)T{z' - /c + 1) by T{z - k)T{z' - k) we see that {z - k){z' - k) 
must be strictly positive for all k But this implies that [z, z') belongs either 

to Zpj-inc or to -Zcompl- 

Thus, we have verified all the claims in the case when both z and z' are noninte- 
gral. Now we shall do the same when at least one of them is integral. By virtue of the 
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2d, 



legen,m — 



{{z, z') E \ z = m, z' > m — 1, or z' = m, z > m — 1}, 



keZ. 



symmetry z <-> 2;', we may assume that 2; = m e Z and z' ^ m — l,m — 2, . . . . Then 
the expression (J^{z—k-\-l)V{z' — k+l))~^ vanishes for k = m+l,m+2, . . . and does 
not vanish for = m, m — 1, . . . . If 2' is real and strictly greater than m — 1 then the 
expression is strictly positive for /c = m, m — 1, . . . , because then both V{z — k + 1) 
and T{z' — k + 1) are strictly positive. Conversely, let T{z — k -\- 1)T {z' — k -\- 1) be 
strictly positive for A; = m, m — 1, . . . . As T{z — k +1) = T{m — k + 1) is strictly 
positive, V{z' — k + 1) must be strictly positive, too (fc = m, m — 1, . . . ). Hence the 
same holds for the ratio T{z' — A; + 1) /T{z' — /c + 2). Therefore, 2' — /c + 1 > for 
all A; = m, m — 1, . . . , which implies z' > m — 1. This concludes the proof. □ 

Proposition 7.10. The set Padm introduced in Definition 7.6 consists of the 

quadruples (2, z, w, w') G V satisfying the following two conditions: 
First, both {z,z') and {w,w') belong to Z. 

Second, in the particular case when both {z, z') and {w, w') are in Z^^^en, o-n extra 
condition is added: let k, I be such that (z, z') e 2^degen,fc o,nd (w, w') e 2^degen,i/ then 
we require k + I > 1 . 

Proof. Let us abbreviate -P^(A) = -P^(A | z, z,w,w'). 

Assume that the above two conditions on (z, z,w,w') G V are satisfied and 
prove that {z, z,w,w') G ^^adm- Indeed, examine in detail the possible cases. If 
none of the parameters is integral then claim (ii) of Lemma 7.9 shows that P'^iX) 
is always strictly positive. If {z,z') G .Zdegen,m while w,w' are nonintegral then 
claim (iii) of Lemma 7.9 (applied to the couple {z, z')) together with claim (ii) 
(applied to the couple {w,w')) show that -Pj^(A) is strictly positive if Ai < m and 
vanishes otherwise. Likewise, if (w, w') G 2degen,m (for some m G Z) and 2, z' are 
nonintegral then -P^(A) is strictly positive when Ajv > —m and vanishes otherwise. 
Finally, if (2, z') G 2^degen,/s and (ty, w') G 2^degen,/ (with some A;, Z G Z) then P'j^{X) is 
strictly positive if A satisfies the inequalities A;>Ai>--->Ajv>— i and vanishes 
otherwise. Since > — / by the second condition, these inequalities are satisfied 
by > 2 different A's. Hence we conclude that in all cases {z, 
required. 

Conversely, assume that {z, z,w,w') G I^adm and prove that the above two con- 
ditions hold. We will verify that {z, z') G Z. Then the similar claim concerning 
(w, w') will follow by virtue of the symmetry property 

P'^{Xi,...,Xn\ z, z, w, w') = P^{-Xn, • • • , -Ai | w, w', z, z'). 

As for the second condition. A; + Z > 1, it follows from the argument above. 
We need two simple lemmas. 

Lemma 7.11. Assume that for a given N and a certain X G GTn the following 
two conditions hold: 

• := (Ai - 1, A2, . . . , Xn) e GTn, i.e., Xi > X2 if N > 2, 

• P'^{X) > and P'{X^) > 0. 
Then 

(^-Ai + 1)(^^-Ai + 1) 
(tu + iV + Ai - 1) (tu' + iV + Ai - 1) 

Proof. Indeed, the above expression coincides with the ratio Pj^{X)/ Pj^^{X^). □ 
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Lemma 7.12. Assume that for a given N and a certain A e GTjv the following 
two conditions hold: 

• AT := (Ai,...,A7v-i,A7v + l) e GTn, i.e., Xn-i >XNifN>2, 

• P;,(A) > and P'{X^) > 0. 
Then 

{w + Xn + l)iw' + Xn + 1) 
(z-Xn + N -l){z' -Xn + N -1) 

Proof. Indeed, the above expression coincides with the ratio P^(A)/P^(AT). □ 

Now we resume the proof of Proposition 7.10. To prove that {z,z') & Z we wiU 
examine in succession four possible cases. 

Case 1: all parameters are nonintegral. Then -P^(A) is always nonzero, hence 
we have Pn{X) > for all N and all A e GTat. Given AT = 1, 2, . . . and /c e Z, 
choose A e GTjv such that Ai = A; and (if A?" > 2) Ai > A2. Then, applying Lemma 
7.11 we get 

{z-k + l){z' -k + l) 
{w + N + k - l){w' + N + k - l)^ ' 

Fix k and let — > 00. Then 

{z-k^l){z' -k^l) ^ {z-k+l){z' -k^l) 
{w + N ^k-l){w' ^k-l)^ ^ ' 

This implies {z — k + l){z' — k + 1) > 0. Since this inequality holds for any /c G Z 
we conclude that {z, z') belongs either to 2^princ or to 2^compi- 

Case 2: at least one of the parameters z, z' is integral and at least one of the 
parameters w,w' is integral, too. Then, using the symmetries z z' and w w' ^ 
we may assume, without loss of generality, that 

z = keZ, z' ^k-l,k-2,...; w = I e Z, w' I - 1,1 - 2, . . . . 

Let N be arbitrary. We have: P^iX) — whenever Ai > or A]v < — L Next, if 
/c>Ai>--->AAr>— / then -Pj^(A) 7^ 0. It follows that k > —I and even more, 
k > —I, because two different A's with -P^(A) > must exist. 

Now we apply Lemma 7.11 toA = {k,k — 1, . . . ,k — 1). The assumptions of the 
lemma are satisfied, and we get the same inequality as in Case 1 above. Moreover, 
as ^ — + 1 reduces to 1, the same argument as above shows that 2;' — A; + 1 > 0, 
i.e., z' > k -1. Hence, {z, z') e ^degen, it- 
Case 3: at least one of the parameters z, z' is integral while both w and w' are 
nonintegral. We may assume that 2; = A; e Z and z' ^ k — l,k — 2, . . . . In this case, 
P'jq{X) does not vanish (and hence is strictly positive) whenever Ai < k. Then we 
apply the argument of Case 2 and get exactly the same conclusion. 

Case 4-' both z,z' are nonintegral while at least one of the parameters w,w' is 
integral. We may assume that w = I & Z and w' 1,1 — 1, . . . . Then we have 
P'jsfiX) > whenever Xn > —l- We apply first Lemma 7.11, where we take any 
A such that Ai = —l + i with i = 1,2,..., and A2 < Ai (if N > 2). The same 
argument as above then gives the inequalities 



{z + l -i + l){z' + 1 -i + 1) > 0, i = l,2,. 
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Next, we apply Lemma 7.12, where we take any A such that Ajv = —I and 
Ajv-i > Ajv {ii N >2). This leads to the inequality 

iw-l + l)iw'-l + l) 
{z + l + N-l){z' + l + N-l) ' i,^,.... 

Applying the same trick as above (N — > oo) we see that the numerator must be 
strictly positive. But then the denominator must be strictly positive for any N. 
This results in the inequalities 

{z + l+j){z' + l+j)>0, j = N-l = 0,l,.... 

Combining these two families of inequalities we get that {z + k){z' + k) > for 
any k E Z, which means that {z, z') belongs either to .Zprinc or to .Zcompi- 
This completes the proof of Proposition 7.10. 

8. Topology on the space of extreme characters 

Recall that there is a 1-1 correspondence between extreme characters of U{oo) 
and points of the set O C M'^°°"'"^, see Theorem 1.3 and Remark 1.6. The aim of 
this section is to prove that this correspondence is a homeomorphism with respect 
to natural topologies on both spaces, the space of extreme characters and the space 
Q,. This result is used in the proof of Theorem 9.1 below, it is also of independent 
interest. 

First, we have to define the topologies in question. Let us start with the space 17. 
We equip it with the product topology of the ambient space M'^°°+^. Since 1^"^°°+^ 
is a separable metrizable space, so is Q,. From the definition of Q, it follows that this 
is a locally compact space. Furthermore, for any positive constant c, the subset of 
the form {u E Vt \ 5^ + 5~ < c] is compact. 

Now let us turn to the space of extreme characters. This is a subset of the 
space X{U (oo)) of all characters. We equip X{U (oo)) with the topology of uniform 
convergence on each of the compact subgroups U (N) C U (oo), N = 1,2,... . Then 
we restrict this topology to the subspace of extreme characters. It is readily seen 
that the topological space thus obtained is separable and metrizable. 

Theorem 8.1. The correspondence u i— x^^^ between the points u; G O and the 
extreme characters of the group U (oo) is a homeomorphism with respect to the 
topologies defined above. 

Proof. Recall that any character x ^ X{U{oo)) is uniquely determined by the 
coefficients P/v(A) of the expansions (7.3), where AT = 1, 2, . . . and A ranges over 
GTjv- It is readily seen that the topology of X{U{oo)) just defined coincides with 
the topology of simple convergence of these "Fourier coefficients". Indeed, the 
crucial point here is that the coefficients are nonnegative and each sum of the form 
^P/v(A), where A ranges over GT^r, equals 1. 

By virtue of the multiplicativity property of formula (1.2), the topology on ex- 
treme characters is defined by convergence of the coefficients -Pi (A), where A e 
GTi = Z. These are simply the ordinary Fourier coefficients of the functions on 
the unit circle in C, which are given by the expression in curved brackets in (1.2), 
i.e., 

F(-\u) = e^"(--^)+7-(--^-^) n ^ + ^V^" - ^ + - , ueC, \u\ = l. 

7=1 1 - - 1) 1 - (« ^ - 1) 

(8.1) 
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According to the remarks above, the claim of the theorem is equivalent to the 
following one: in the set of functions of the form (8.1), the uniform convergence 
on the unit circle (or, which is the same, the simple convergence of the Fourier 
coefficients) is equivalent to the convergence of the labels uj in the space VL. Let 
us notice once again that the reformulation in terms of the convergence of the 
Fourier coefficients is possible because the functions (8.1) are positive definite and 
normalized at tt = 1. 

We proceed in a few steps. 

Step 1. Let us prove that the map uj i— > F^'^^ is continuous. Rewrite (8.1) in the 
form 

t}i (l-a+(«-l))e«^(«-i) (l-ar(^,-i-l))e«i ' 

(8.2) 

where we used the fact that 7^ = 5^ — Yli^f + f^t)- suffices to show that the 
infinite product in (8.2) converges uniformly on u and (a"'", /?"'", a", /3~), where u 
ranges over the unit circle and the sum of the parameters af, pf" is bounded by 
a constant. This is reduced to the following elementary fact: the infinite product 
Yli^ + Xi) exp(— Xi) converges uniformly on any subset of C°° of the form {{xi) G 
C°° I ^ < c}, where c is an arbitrary positive constant. 

Note that without exponentials, the convergence is nonuniform. For this reason, 
we cannot take 7=*= instead of 5"^. Note also that 7+ and 7" are not continuous 
functions of u>. 

Step 2. For ou E 0,, set = + 5~ . We claim that the inverse map F^^^ — > cj 
is continuous provided that u) is subject to the restriction ||a;|| < c, where c is an 
arbitrary positive constant. Indeed, this follows from the result of step 1, because 
any subset of the form \\u\\ < c is compact. Here we use the fact that a bijective 
continuous map of a compact space on a Hausdorff space is a homeomorphism. 

Step 3. For a; e fi, set 

ii^ir = 7++7-+E«^^(i+«.^)+EA+(i-/^^^)+E«^"(i+«^")+EA"(i-/5r)- 

Note that all summands are nonnegative and the sums are finite. Let (cOn) be a 
sequence of points of O such that the corresponding sequence of the functions F^'^"-^ 
is convergent. We claim that then remains bounded. 

Indeed, recall that any function of the form (8.1) is positive definite on the unit 
circle and normalized at 1. Hence, it is the characteristic function (i.e., Fourier 
transform) of a probability measure on Z. Since (8.1) is a real analytic function, 
the corresponding measure possesses finite moments of any order. Note also that 
the uniform convergence of characteristic functions on the unit circle is equivalent 
to the weak convergence of the corresponding probability measures. 

We will need the following 

Lemma [OkOl, Lemma 5.2]. Let (M„) be a sequence of probability measures on 
Z {or, even more generally, on M) such that each has finite moment of order 
4 and the sequence (M„) weakly converges to a probability measure. Assume that 
the second moments of 's tend to infinity. Then the fourth moments grow faster 
than the squares of the second moments. 

Actually, we will use a corollary of the lemma. 
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Corollary. Let (M^) be a sequence of probability measures on Z with finite 4-th 
moments, weakly convergent to a probability measure. Assume that for any n, the 
4-th moment of M„ is bounded by the square of the 2-nd moment times a constant 
which does not depend on n. Then the 2-nd moments are uniformly bounded. 

Consider the functions Gn{u) = These are also positive def- 

inite normahzed functions on the unit circle, and the sequence (G^) is uniformly 
convergent by the assumption on the initial functions. Let be the probabil- 
ity measures corresponding to the functions Gn- Then the measures M„ weakly 
converge to a probability measure. We will prove that these measures obey the as- 
sumption of the corollary, which will imply the uniform boundedness of their second 
moments. This, in turn, will imply that ||a;n||' remains bounded, as required. ^ 

Let us realize this plan. Set it = e*^. If M is a probability measure on Z and 
G{u) = G(e*^) is its characteristic function, then the 2-nd and 4-th moments of M 
are equal, within number factors, to the 2-nd and 4-th coefficients of the Taylor 
expansion of the function ^(e*^), respectively. 

Given a; e fi, introduce new variables as follows 

{a,} = {a+(l+a+)}u{a^(l+a^)}, {b,} = {f3+ (l-f3+)}u{f3^ {l-f3^)}, c = ^++^-. 

Here the ordering of the variables a^, bi is unessential. In this notation we get, after 
a simple computation. 



pM^P^^^i?(c.)fe^e^ _ g2c(cose-i) TT 1 + 26^(cosg- 1) 
[e )-t [e )-e 11 1 - 2a,(cos ^ - 1) 

a2 , 1 , -TT 1 — biQ'^ -\- T^biO^ + . . . 



In this expression, the coefficient of equals 

-{c + Y.ai + Y.bi)^-\\u:\\' (8.3) 
and the coefficient of equals 

^(c + ^ aj + ^ 6i) + ^c^ + ^ a1 + 62(0, ai, 02, . . . , 61, 62, . . • ), (8.4) 

where 62 stands for the 2-nd elementary symmetric function. It is readily seen that 
the expression (8.4) is bounded from above by the square of the expression (8.3) 
times a constant. This shows that in our situation, the 4-th moment is always 
bounded by a constant times the square of the 2-nd moment. As was explained 
above, this implies that the second moments remain bounded. By virtue of (8.3) 
this exactly means that ||ci;n||' remains bounded, as was required. 

Step 4- Let a sequence {uJn) be such that (F*^"^"^) is convergent. Here we will show 
that \\OJn\\ remains bounded. By virtue of step 2 this will imply that {uJn) converges 
in As we noticed in the very beginning of the section, our topological spaces are 

^We have replaced the initial functions by the squares of their moduli, because this simplifies 
the estimation of the moments. 



37 



separable and metrizable. Therefore, it will follow that the inverse map F^'*') i— > uj 
is continuous, which will complete the proof of the theorem. 

By virtue of step 3, remains bounded. We would like to deduce from this 

that llo^nll remains bounded, too. 

Let us compare \\uj\\ and \\uj\\' ■ We have 

af<af{l + af) 

and 

< 2/?±(l - provided that e [0, \]. 

In general, we only know that the coordinates are in [0, 1], so that there is no 
universal estimate of the form ||a;|| < const - WojW . We will bypass this difficulty as 
follows. 

For any G O, only finitely many coordinates (3^ or [3~ are strictly greater than 
|. Let us call these coordinates "bad", and let K{u)) denote their number. Then 
we have an estimate of the form ||a;|| < C ■ where C depends only on K{ijj). 

Now it suffices to show that K{ujn) remains bounded, which will imply that 
remains bounded. 

Recall that for any a; G (3i + /3f ^ 1- Hence, both (3^ and (5^ cannot be 
"bad". Therefore, if K{ijj) > 0, then the "bad" coordinates are either the first K{lij) 
coordinates of /?"'' or the first K{lij) coordinates of P~ . Define a new element uj e ^ 
as follows. If the "bad" coordinates were in Z?"*" then we remove them, then add to 
(3~ new coordinates 1 — (3^ , . . . , 1 — and finally rearrange all coordinates in 

f3~ in the descending order. If the "bad" coordinates were in /3~ then we do the 
same operation with (3^ and (3~ interchanged. Then we have 

K(u) = 0, = ||a;||'. (8.5) 

On the other hand (see Remark 1.5), 

F^^\u) = F^'^\u)u'^^^'^K (8.6) 

Let us assume now that in our sequence (a;„), the quantity K{u!n) is unbounded. 
Taking a subsequence, we may assume that K{un) — > oo. Consider the correspond- 
ing sequence (cJ^). Since \\ujn\\' remains bounded and \\uJn\\' = \\i^n\\' (see (8.5)), 
we conclude that \\uJ^\\' is bounded, too. Since does not have "bad" coordinates 
(see (8.5)), the argument above shows that ||cl^^|| remains bounded. It follows that 
the corresponding functions on the unit circle form a relatively compact set in the 
topology of uniform convergence. Taking a subsequence again, we may assume that 
the functions F*^^"^ converge. On the other hand, by the initial assumption, the 
functions F^^"-^ converge, too. Now taking account of (8.6) we obtain a contra- 
diction. Indeed, we get two sequences of continuous functions on the unit circle, 
normalized at 1, say Fn{u) and Fn{u), such that 

Fn{u) = Fn{u)u^^'^ , where Kn ^ oo, 

and such that both {F^} and (Fn) converge in the uniform metric. These conditions 
imply that the ratios 

Fn{u) 
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uniformly converge in a neighborhood of tt = 1, which is impossible, because — > 
oo. 

This contradiction shows that actually K{u!n) must be bounded, which completes 
the proof. □ 

9. Existence and uniqueness of spectral decomposition 

The aim of this section is to rederive (and slightly refine) a result due to Voiculescu: 

Theorem 9.1 (Cf. [Vo, Theoreme 2]). For any character x of the group U{oo) 
there exists a probability measure P on the topological space Q such that 

X{U) = [ x^'^\U)P{dw), U e U{oo). 
Jo. 

Moreover, such a measure is unique. 

We call P the spectral measure of x- 

Comments. 1. Recall (sec the beginning of §8) that O is a good topological space 
(locally compact, separable, metrizable), so that there is no problem in defining 
measures on it. Specifically, we take Borel measures with respect to the natural 
Borel structure of Q,. 

2. The integral above makes sense, because x^'^\U) is a continuous function in 
to (see Theorem 8.1) and \x^'^\U) \ < 1. 

3. It is readily seen that for any probability Borel measure P on the above 
formula defines a character, so that the correspondence x ^ -P is a bijection between 
the space X{U{oo)) of characters and the space of probability Borel measures on 
O. 

4. In Remark 9.4 below we compare our approach with that of Voiculescu. 

We shall derive Theorem 9.1 from a more general claim, see Theorem 9.2 below. 

Assume we arc given a sequence Fq, Fi, ... of nonempty sets, where Fq con- 
sists of a single point denoted by the symbol while each F^r with > 1 is a 
finite or countable set. Let Ajv denote the space of formal convex combinations of 
the points of Fat; this is a simplex whose vertices are points of Fjv- Further, as- 
sume we are given a function g(z/. A) defined on couples (z^. A) G F^r-i x F^v, where 

= 1, 2, . . . , such that < g(z/, A) < 1 and, for any A G F^r, Yin, q{^j ^) = 1- 
the particular case A^ = 1 this means that q{0, A) = 1 for all A G Fi. 

For each A" = 1,2,..., there exists a unique affine map An ^ An-i taking any 
A G Fjv to the convex combination li^^ ^ An-i. Let A = ^m Ajv denote 
the projective limit taken with respect to these affine maps. 

Let F = Fq U Fi U F2 U . . . be the disjoint union of the sets F^v; this is a countable 
set. Denote by JF the vector space of all real functions on F and equip it with the 
topology of pointwise convergence on F. Note that J-' is a locally convex vector 
space. It is metrizable, because F is countable. 

We identify A with the subset of formed by nonnegative functions / such that 
/(0) = 1 and f{iy) = XIa Qi^^ ^)fi^) for any u G Fjv-i, AT = 1, 2, ... , where the 
summation is taken over A G Fjv (note that these conditions imply that /(A) = 
1, summed over A G Fjv)- 
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Clearly, A is a convex subset of T. As a subset of A inherits its topology. We 
also consider the Borel structure on A generated by this topology. One can prove 
that this structure is standard. 

Let the symbol Ex( ■ ) denote the subset of extreme points of a convex set. 

Theorem 9.2. Let A fee ihe convex set |im A^r defined above. The subset Ex(A) of 
extreme points of IS. is a Borel subset and each point f E A is uniquely represented 
by a probability Borel measure P on Ex(A).- / = J^_^^^-^ gP{dg). That is, /(A) = 

/ex(A) 9WP{dg) for any A e T. 

Derivation of Theorem 9.1 from Theorem 9.2. Take Fat = GTat, = 1, 2, . . . and 
take as qiy^X) the cotransition probability function (Definition 7.2). Then Ajv 
turns into the set of probability measures on GTjv and the set A becomes the set 
of coherent families of measures on GT (Definition 7.3). By virtue of Proposition 
7.4 we get a bijection between A and X{U{oo))^ which is an isomorphism of convex 
sets. Moreover, this is a homeomorphism of topological spaces (see the discussion 
of the topology on X{U{oo)) in the beginning of §8). Applying Theorem 1.3, 
which provides an explicit parameterization of extreme characters, we get a bijective 
correspondence between Ex(A) and Q,. By Theorem 8.1, this correspondence is a 
homeomorphism of topological spaces. Hence, it preserves the Borel structures. 
This turns Theorem 9.1 into a special case of Theorem 9.2. □ 

The proof of Theorem 9.2 is based on the following lemma. 

Lemma 9.3. The convex set A is a Choquet simplex, i.e., the cone generated by 
A is a lattice. 

Proof. Let C denote this cone. It coincides with the subset of that is described 
similarly to A C .7-": the only difference is that the condition /(0) = 1 is dropped. 

We extend the function q{u, A) to any couples f £ Tm, A G Fjv with M < N 
using the following recurrence relation: 



Given /i , /2 G C we construct their lowest upper bound as follows. Define a 
function / on F by 



The limit exists, because, for any fixed z/, the A^th sum monotonically increases as 
N ^ oo and remains bounded from above by fi{i') + /2(z^)- It is readily verified 
that / belongs to the cone and is the lowest upper bound for /i and /2. 

The existence of the greatest lower bound is verified similarly: it suffices to 
substitute "min" for "max". □ 

Proof of Theorem 9.2. If all sets Fjv are finite, then A is compact. Since A is 
metrizable (as J- is metrizable), the claims of Theorem 9.2 immediately follow from 
Lemma 9.3 and Choquet's theorem, see [Ph]. When the sets F^v are allowed to 
be countable, the space A may be noncompact, so that Choquet's theorem is not 
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pGrjv_i 



f{iy)= \im g(z/,A)max(/i(A),/2(A)). 



directly applicable. To overcome this difficulty we embed A into a bigger set A C 
which is compact. 

Specifically, let A be the set of nonnegative functions / on F such that f{0) = 1 
and f{u) > '^x'ii^^ -^)/('^) ^ ^ Tn-i, = 1,2, . . ., where A ranges over 

Fn (i.e., in the definition of A, we have replaced the equality by an inequality). 
We note that A is a compact convex set containing A. 

Next, for any M = 0, 1, ... , let Am be the set of nonnegative functions on F 
such that /(0) = 1, f{i^) = Ea?(^'^)/(^) for aU e F^v-i with N < M, and 

/( ■ ) = on Fm+1 LJFm+2 U Note that Am is a convex subset of A, isomorphic 

to Am- 

Finally, dropping the condition f{0) = 1 in the definitions above we get the 
cones spanned by the sets A and Am; we denote them by C and Cm, respectively. 

Now the crucial remark is that any element / e C is uniquely represented as a 

sum / = /o + /i H h /oo, where /m e Cm and foo e C. Conversely, any such 

sum represents an element of C provided that fM{^) < +oo, cf. [013, §22]. 

This implies a number of consequences. First, the cone C is a lattice, so that 
we may apply Choquet's theorem to A. Next, Ex(A) is the disjoint union of the 
sets Ex(Am) ^ Fm, where M = 0, 1, . . . , and the set Ex(A). Thus, Ex(A) is the 
difference of Ex(A), which is a Borel set, and a countable set; whence, Ex(A) is 
a Borel set. Finally, if / belongs to A then its representing measure on Ex(A) is 
concentrated on Ex( A) C Ex( A) . This concludes the proof. □ 

Remark 9.4. Here is a comment to Voiculescu's original result [Vo, Theoreme 2]. 
Its formulation says that any x £ X{U{oo)) is uniquely represented by a probability 
measure P on the space Ex{X{U (oo))), equipped with the Borel structure inherited 
from the ambient space X{U{oo)). To prove the existence of P, Voiculescu also 
embedded X{U{oo)) into a compact space. But he used a special property of the 
cotransition function g(z/. A) of the graph GT (specifically, the fact that q{iy, A) tends 
to zero when v e GTjv_i is fixed and A e GTn goes to infinity). This allowed him 
to take, instead of our space A, a smaller set. Voiculescu's proof of the uniqueness 
statement is quite different from ours: it substantially relies on the multiplicativity 
property of extreme characters. The argument presented above seems to be more 
general and direct. 

Voiculescu's original formulation did not involve the space O., because at that 
time it was not yet clear whether the characters x^'^^ exhaust the whole set Ex{X{U{oo))) 
of extreme characters. Our (modest) supplement consists in checking that the bi- 
jection between the spaces Ex{X{U{oo))) and O preserves the Borel structures, so 
that P can be carried over from the "abstract" space Ex(A'(?7(oo))) to the "con- 
crete" space Q. Of course, this is a pure technical claim whose validity seems to be 
beyond doubt. However, it is not completely trivial. 

10. Approximation of spectral measures 

The aim of this section is to establish a relationship between the spectral measure 
P of an arbitrary character x and the coherent system {Pn} corresponding to %. 
Recall that Pn is a probability measure on the discrete set GTjv C (iV = 
1,2,...) while P is a probability Borel measure on the region O C R^°°"'"^ (see 
Proposition 7.4 and Theorem 9.1). We will prove that, as — oo, the measures 
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Pjv approximate the measure P in a certain sense. To precisely state this claim we 
need a few definitions. 

Given a signature A G GT^v, we denote by A"*" and A~ its positive and negative 
parts. These are two Young diagrams such that ^(A"*") + ^(A~) < N, where £{■) is 
the number of nonzero rows of a Young diagram. That is, 

A = (A+, . . . , A+, 0, . . . , 0, -Ar, . . . , -Ar), k = e{x+), I = e{x-). 

Next, given a Young diagram u, we denote by d{u) the number of diagonal boxes 
in v, and we introduce the Probenius coordinates of v: 

Pi{v) = Vi-i, q^(u)^u--i, i = 1, . . . , d{u), 

where v' stands for the transposed (conjugate) diagram. 

Then, following Vcrshik-Kerov [VKl], we introduce the modified Frobenius co- 
ordinates of v as follows 

Pii^^) = Pii^^) + ^ ^ - i + h Qii^) = Qii^) + ^ = - ^ + h i=l,...,d{u). 
Note that 

Pi{u) > ■■■ >Pd{u){i^) > 0, gi(z^) > ■■■ > gd(^)(z^) > 0, 



i=l 

where |z^| denotes the total number of boxes in u. We also agree to set 

Pi{j^) = = 0, i = d{i') + 1, d{i') + 2, . . . , 

so that the coordinates Pi(z^) and are now defined for any i = 1, 2, . . . . 
Definition 10.1. For any A?" = 1, 2, . . . , we embed the set GTjv into as follows 

GTjv 9 A I — > CO = (o"*", 6"*", a~ , b~ , c"*", c~) G n, 

"'i — AT- ' — AT- I* — -L) ^) • • • 



N ' ' AT ^ ' ' ^' AT 

Theorem 10.2. Let x be an arbitrary character ofU{oo), P be its spectral measure, 
and {Pn} be the coherent system corresponding to x- Further, for any N , convert 
Pn to a probability measure Pj^ on O, which is the pushforward of Pn under the 
embedding GTn ■— O introduced in Definition 10.1. 

Then, as N ^ oo, the measures P^ weakly converge to the spectral measure P. 
I.e., for any bounded continuous function F on Q, 



lim / F{uj)Pj^{dw) = [ F{u))P{dw). 



The proof is quite similar to that of [BOS, Theorem 5.3]. We will see that 
Theorem 10.2 is a corollary of another general result. Theorem 10.7. 
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A path in the Gelfand-Tsetlin graph GT is an infinite sequence t = {ti,t2, ■ ■ ■) 

such that tN e GTn and tjv -< tN+i for any N = 1, 2, The set of the paths 

will be denoted by T. 

We also need finite paths. A finite path of length is a sequence t — (ri, . . . , tn), 
where n G GTi, . . . , tn G GTjv and ti -< • • • ^ tat. The set of finite paths of 
length N will be denoted by Tjv- One can identify T with the projective limit 
space ^mTJv, where the projection 7jv — > Tkr-i is the operation of removing the 
last vertex ttv- 

Consider the natural embedding T C H GT^r. We equip Yl GTjv with the prod- 

N N 

uct topology (the sets GTn are viewed as discrete spaces). The set T is closed 
in this product space. We equip T with the induced topology. Equivalently, the 
topology is that of the projective limit space ^mT/v- Then T turns into a totally 
disconnected topological space. 

Given a finite path r = (ti, . . . , tjv) G 7/v, define the cylinder set d T as the 
inverse image of {r} under the projection T Tn , 

Cr = {t e T \ ti = Ti, . . . ,tN = Tn}- 

The cylinder sets form a base of topology in T. 

Consider an arbitrary signature A G GTat. The set of finite paths r = (ri ^ 
••• -< Tn) ending at A has cardinality equal to Dim^v A = X^l^)- The cylinder 
sets Cr corresponding to these finite paths r are pairwise disjoint, and their union 
coincides with the set of infinite paths t passing through A. 

A central measure is any probability Borel measure on T such that the mass of 
any cylinder set Cj- depends only of its endpoint A. Clearly, central measures form 
a convex set. 

These definitions are inspired by [VKl]. 

Proposition 10.3. There exists a natural bijective correspondence M < — > {Pn} 
between central measures M and coherent systems {Pn}, defined by the relations 

DimAT A • M(C^) = Pjv(A), (10.1) 

where N = 1,2, . . . , AG GTtv, and r is an arbitrary finite path ending at A. 

In other words, the relations mean that for any N , the pushforward of M under 
the natural projection 

oo 

GTjv D T ^ GTat (10.2) 

coincides with Ptv- 

Proof. Let {Pn} be a coherent system. For any N , we define a measure Mn on 
the discrete space Tn as follows. Given r G TJv, we set 

where A G GTtv is the end of r. This is a probability measure. Its pushforward 
under the projection Tn —>■ T/v-i coincides with Mn-i' indeed, this is exactly a 
reformulation of the coherency property (7.2). Hence the family {Mn} determines 
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a probability measure M on the projective limit space T. Clearly, M is a central 
measure, and we have the relations (10.1). 

Conversely, let M be a central measure. For any A^, define a probability measure 
P/v on GT isf as the pushforward of M under the projection (10.2). The fact that M 
is central then implies that the family {Pn} satisfies the coherency property. □ 

Corollary 10.4. There is a natural bijective correspondence x ^ — ^ M between 
characters and central measures. This correspondence is an isomorphism of convex 
sets. 

Proof. The bijection is established by means of the bijections x ^ — ^ {Pn} (Propo- 
sition 7.4) and {Pn} < — ^ M (Proposition 10.3). From the proofs of these proposi- 
tions it is clear that this is an isomorphism of convex sets. □ 

Definition 10.5. Let t — (^tv) G T be a path. We say that t is regular if the 
images of the tjv's under the emeddings GTjv ^ ^ (see Definition 10.1) converge 
to a point u> & Q. Then uj is called the end of t. Equivalently, the condition means 
that there exist limits 



<=i™^^' /^f=Ji^^^ (z = l,2,...), 5^=lim 



where and t]^ denote the positive and negative parts of tjv £ GTn. The set of 
regular paths will be denoted by T^eg- 

Introduce the map 

TT : T^eg ^, %.eg ^ 1 1 — > {the end of t} e Q. (10.3) 
Further, for any A?" = 1, 2, . . . , we introduce the map ttn : T Q as the composition 

^ ti-^tjv ^rrp Definition 10.1 ^ ,n 

TTn '■ ^ >■ GTjv > ii. (10-4) 

For any t e ^eg, '^N{t) converges (as N — > oo) to Tv{t) in the topology of the space 
fl. Indeed, this holds by the very definition of regular paths, see Definition 10.5. 

Lemma 10.6. T^eg dT is a Borel set, and the maps n, tin are Borel maps from 

j'gg to • 

Proof. Indeed, for any N, the functions 



P.itN) t^mi (^ = l,2,...), t^l' 



AT ' AT ^ ' ' ^' N 

are continuous functions on the space T. This implies that the regularity condition 
determines a set of type F^s, hence a Borel set. 

Each TTn, being a cylindrical map, is continuous on T. Its restriction to T^eg is 
also continuous, hence Borel. Finally, tt is the pointwise limit of the ttjv's, hence it 
is a Borel map, too. 
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Theorem 10.7. Any central measure M is concentrated on the set T^eg C T. The 
pushforward tt{M) under the projection (10.3) coincides with the spectral measure 
P of the character x that corresponds to M. 

This claim makes sense, because T^eg is a Borel set. 

Derivation of Theorem 10.2 from Theorem 10.7. Let M be the central measure 
corresponding to x- 

By Proposition 10.3, we have 7r7v(M) = P^. Now we will interpret M as a 
probability measure on T^eg, which is possible by virtue of Theorem 10.7. Also, tin 
will be viewed as a projection — > Q,. Then the equality 7rjv(M) = remains 
true. On the other hand, 7r(M) = P (again by Theorem 10.7). Thus, both P^ and 
P can be viewed as pushforwards of one and the same probability measure M. 

Now let P be a bounded continuous function on VL. Since iTNit) —>■ 7r(t) for any 

t G Tj-Qg, 

FiTTNit)) ^ P(7r(t)), t e Treg. 

Thus, we get a sequence {F{tvn{-))} of uniformly bounded functions on T^eg con- 
verging pointwise to the function P(7r( • )). All these functions are Borel functions, 
because ttjv and tt are Borel maps (Lemma 10.6). Consequently, as A?" — > oo. 



i 



F{nN{t))M{dt) — > I F{n{t))M{dt). 



Since 7rjv(M) = P^y and 7r(M) = P, we can convert these integrals to integrals over 

n, 

I F{nN{t))M{dt) ^ I F{uj)PN{dio), j F{n{t))M{dt) ^ j F{io)P{dio), 

which concludes the proof. □ 

The rest of the section is devoted to the proof of Theorem 10.7. 

Given two signatures, u e GT„ and A e GTn, where n < N, denote by 
DimnAr(z^, A) the number of paths starting at v and ending at A. Le., the num- 
ber of chains 

T = (t„ ^ Tn+l -< ■■ ■ ^ Tn-1 -< Tat), Tn = l^, Tn = A. 

We extend the definition of the cotransition function by setting (cf. the proof of 
Lemma 9.3) 

Dimn(z^)DimnAr(j^, A) \ ^ rn^ ^ at 

qiv, A) = , v e GT„, A e GTjv, n < N. 

DimAT A 

Note that 

AT 

T i=n+l 

summed over all r's as above. 

For any coherent system {Pat}, we have 

Pn{iy)^ J2 ?(i^,A)Piv(A), ueGTn, n<N. 

Indeed, this relation is obtained by iterating the coherency relation (7.2). 

45 



Proposition 10.8. Let {Pn} be a coherent system and M be the corresponding 
central measure. Assume that M is extreme. Then for M-almost all paths t = 

(tN) e r 

lim q{u,tN) ^ Pn{T^) , n = l,2,..., ueGT^. (10.5) 

AT— »oo 

Comment. This result actually holds for general "branching graphs" in the sense 
of Vershik-Kerov [VK3]. It was stated in their paper [VKl] for the Young graph, 
associated with the infinite symmetric group S'(oo). A detailed proof (an adap- 
tation of the proof of the Birkhoff-Khinchine ergodic theorem) is contained in an 
unpublished work by Kerov. We present below a similar argument but use a limit 
theorem for reversed martingales (as was suggested in [VKl]). Note also a very 
close earlier result due to Vershik, see [Ve, Theorem 1] and [OV, Theorem 3.2 and 
Remark 3.6]. 

Proof. Step 1. Let us say that two paths t = {tm)i t' = {t'^) are N -equivalent if 
tm = t'm m> N. Denote by ^Ar(t) the A'"-equivalence class containing t. Then 

{t} = 6(t)c6(t)c.... 

We say that t and t' are oo -equivalent if they are A^-equivalent for large enough. 
The oo-equivalence class of t is denoted by ^oo{t)- Clearly, Coo{t) — U^Af(t). 

Let B_jv denote the u-algebra of those Borel sets in T that are saturated with 
respect to the A-equivalence relation, where N = 1,2,... . We define B-oo likewise. 
We have 

oo 

B-oo C • • • C i3_2 C = {All Borel sets}, B-oo = Q B-n 

N=l 

Fix an arbitrary bounded Borel function tfj on T. We will view ifj as a, random 
variable defined on the probability space (T, M). Let '0jv = ^{i^ \ B-n) and 
ipoo = ^{fp I -S-oo) be the conditional expectations of ip with respect to the a- 
algebras B-n and B-oo, respectively. 

By the theorem on convergence of (reversed) martingales, we have 

lim ipN = V'oo almost everywhere with respect to M, 

N^oo 

see, e.g., Doob [Do, Chapter VII, Theorem 4.2]. 

Step 2. So far we did not use the assumption that M is central. Now we 
remark that this assumption makes it possible to describe each tpN explicitly as the 
averaging over the A"-equivalence classes: 

Next, we use the assumption that M is an extreme central measure to conclude 
that each set A G B-oo has mass or 1. Indeed, assume < M{A) < 1, and let 1a 
stand for the characteristic function of A. Then the measure M can be written as 
a nontrivial convex combination of two probability measures, 

M = MiA).^^+il-MiA)) 



M{A) ' ' ''1-M{A) 
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Since A is saturated with respect to the oo-equivalence relation, these two measures 
are central, which contradicts the extremality assumption. 

It follows that tfjoo is constant almost everywhere. Since the means of all V'iv's 
are the same and equal to the mean of if), the constant is the mean of t/j. 

Therefore, we conclude: let ^p be any bounded Borel function on T and let i^Nit) 
be defined by (10.6); then 

lim Viv(t) = / i^{t')M{dt') for M-almost all t e T. (10.7) 

Step 3. Fix n and i/ e GT^^, and set 

m = 



1, if tn = V, 

0, otherwise. 



For any N > n, we have 



Diiii„,(;/) Dim„,:v(//, ^at) 
^^^'^ = = 

Indeed, the set (t) contains exactly Dim„(z^) Dim^jv(i^, ^at) paths t' passing through 
v. Finally, the mean of '0 equals Pn{j^)- 

Now we apply the result of Step 2 and get from (10.7) the required claim. □ 

Assume that for any = 1,2,... we are given a character (in the sense of 
Definition 1.1) of the group U{N), denoted as x^''^^- Let us say that the sequence 
{x^^^} converges to a function defined on the group U(oo) if 

as A ^ oo, x^^^ \uin) X^°°^ \uin) , for any fixed n = 1, 2, . . . (10.8) 

Clearly, the limit function x*^""-* is a character of U (oo). Expand irreducible 
normalized characters of U{N): 



AeGTjv 



Then, for any n < N, 



= E f E g(-,A)p(^nA))x^ 



It follows that (10.8) is equivalent to the following condition: for any n and any 

hm g(^,A)P^^)(A)=Pn(^), (10.9) 

AT— »oo ' 
AeGTiv 

where {-Pat} is the coherent system corresponding to x*'°°''- 
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Proposition 10.9. Let {A(A'") e G'irjv}jv=i,2,... be a sequence of signatures and let 
— x^^^^ be the corresponding normalized irreducible characters. 
The sequence {x*-^-*} converges to a function x^^^ ^/ ^'^^ '^''^^U ^/ images of 
the X{N) 's under the embeddings of Definition 10.1 converge to a point uj and 
then the limit function x*-""-* coincides with the extreme character x^'^^ ■ 

Proof. This result is due to Vershik-Kerov [VK2] . For a detailed proof, see [OkOl] . □ 

Proposition 10.10. Let u E and M^^^ be the extreme central measure corre- 
sponding to the extreme character x^'^^ . The measure M^^^ is concentrated on the 
subset Tr~^ (uj) C %eg of regular paths ending at u. 

Proof. By Proposition 10.8, the measure M*^"^^ is concentrated on the set of paths 
i — {^n) G ^ satisfying the condition (10.5), where {-Pat} is the coherent system 
corresponding to the character x^'^^- Let us show that this set coincides with 

Indeed, the condition (10.5) coincides with the condition (10.9) for the characters 
x'-'^-* = X*^ (here p(^) is reduced to the delta measure at tj^)- As explained above, 
the latter condition is equivalent to the convergence of the characters x*^^'* to the 
character x*^'^'*- By Proposition 10.9, this exactly means that t is a regular path 
ending at a;. □ 

Proof of Theorem 10.7. Translating Theorem 9.1 into the language of central mea- 
sures we get the decomposition 

M = [ M^'^^Piduj). 
Jo. 

By Proposition 10.10, any extreme measure M^'^^ is concentrated on the subset 
7r~^(a;) C %eg,- Hence, M is concentrated on T^eg- This proves the first claim of 
Theorem 10.7. The second claim also follows from the above decomposition and 
Proposition 10.10. □ 

11. Conclusion: the problem of harmonic analysis 

Now we are in a position to state the problem of harmonic analysis on the group 

U{oo): 

Let (^, ^, w, w') G I^adm bc an arbitrary admissible quadruple of parameters 
(Proposition 7.10), let Pn = {Pn{- \ z, z,w,w')}n=i,2,... be the coherent system 
as defined in (7.8), let Xz,z,w,w' be the corresponding character, and let P be the 
spectral measure of Xz,z,w,w' ■ 

Describe explicitly the measure P. 

Theorem 10.2 suggests the idea to evaluate P by means of the limit transition 
from the measures Pjy- This idea is realized in the subsequent paper [B04]. 

Let X be an arbitrary character of U{oo) and T^ be the corresponding spherical 
representation of {G,K), see §2. One can prove that the spectral measure of x 
also determines the decomposition of into a (multiplicity free) direct integral 
of irreducible spherical representations. Recall that in §3, we constructed a family 
{Tzw} of unitary representations. If 3?(z + w) > — | then T^w possesses a distin- 
guished vector, and if, moreover, both z and w are nonintegral, then this vector is 
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cyclic (Proposition 6.9), so that T^w coincides with the representation with the 
character x = Xzw Thus, in this case, the spectral measure of the character Xzw 
also governs the decomposition of T^w 

If 'Si{z + w) > — ^ but at least one of the parameters z^w is integral, then the 
spectral measure of the character Xzw refers to a proper subrepresentation of Tzw 
To decompose the whole representation T^w we need additional tools. Cf. [KOV]. 

Finally, recall that the construction of the representations Tzw makes sense even 
when 'Si{z + w) < — |, although then the characters Xzw disappear. It would be 
interesting to study the decomposition problem in this case as well. Cf. a similar 
problem concerning infinite measures m^^^ stated in [B03, §8]. 
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